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ABSTRACT 


The purpose of this thesis is to provide a methodological approach for conducting 
a Business Case Analysis (BCA) for the Advanced Technology Ordnance Surveillance 
(ATOS) Advanced Concept Technology Demonstration (ACTD). This study provides a 
methodology for comparing the cost savings over time of having an ATOS infrastructure 
in place at Navy Munitions Management locations compared to the base case of “as-is” 
inventory management. 

ATOS is a Radio Frequency Identification (RFID)-based automated system that 
provides a capability to collect environmental data in near real-time and supports 
munitions management tasks on stored munitions pallets. This type of data has never 
before being available and is critical for Ordnance Managers and warfighters in making 
more effective decisions about the shelf life and operational performance of individual 
munitions throughout their service life. ATOS is not meant to be a replacement for the 
status quo processes of Ordnance Management (OM). Instead, ATOS is meant to 
enhance many of the current processes and add additional capabilities and dimensions to 
OM. 

A Business Case Analysis for a notional site, using exemplar data sets indicates 
that with an initial investment of $1.3M, a Net Present Value (NPV) of $5.3M can be 
obtained over a 10-year period. The payback period is less than one year, and the Return 


on Investment is 214 %, or almost 11 % annual, compounded ROI. 
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EXECUTIVE SUMMARY 


The purpose of this thesis is to provide a methodological approach for conducting 
a Business Case Analysis (BCA) for the Advanced Technology Ordnance Surveillance 
(ATOS) Advanced Concept Technology Demonstration (ACTD). This study provides a 
methodology for comparing the cost savings over time of having an ATOS infrastructure 
in place at Navy Munitions Management locations compared to the base case of “as-is” 
inventory management. 

ATOS is a Radio Frequency Identification (RFID)-based automated system that 

e Provides a capability to collect environmental data in near real-time, and 
e Supports munitions management tasks on stored munitions pallets. 

These types of data have never before being available and are critical for 
Ordnance Mangers and warfighters in making more precise predictions about the shelf 
life and operational performance of individual munitions throughout their service life. 
ATOS is not meant to be a replacement for the status quo processes of Ordnance 
Management (OM). Instead, ATOS is meant to enhance many of the current processes 
and add additional capabilities and dimensions to OM. 

With the assistance of Subject Matter Experts (SMEs) from Naval Surface 
Warfare Center (NSWC), Indian Head Division, a Work Breakdown Structure (WBS) 
with Cost Estimates (CEs) for the implementation of an ATOS infrastructure was 
generated. These cost estimates were used in calculations to determine the “as-is” and 
the “to-be” inventory costs. These costs were used in a Business Case Analysis to 
determine the potential cost savings and benefits of implementing an ATOS 
infrastructure at a notional, five magazine munitions site, which is meant to be analogous 
to Navy Munitions Management locations. 

This Business Case Analysis methodology only provides cost savings for this 
notional base case. In order to provide realistic cost savings with a greater degree of 
accuracy, a pilot program has been scheduled for later this year to assess further the 


benefits, including cost impacts, of the ATOS technology. 


XVil 


A Business Case Analysis for a notional site, using exemplar data sets indicates 
that with an initial investment of $1.3M, a Net Present Value (NPV) of $5.3M can be 
obtained over a 10-year period. The payback period is less than one year, and the Return 
on Investment is 214 %, or almost 11 % annual, compounded ROI. 

To validate the cost estimates used, a full scale ATOS pilot project at a munitions 
location, such as, Seal Beach, needs to be funded and implemented in order to collect the 
required data to validate the results from previous analysis, and support and provide a 
better argument as to why ATOS is the way to go in Ordnance Management. 

Additionally, in order to establish an ATOS implementation plan DoD wide, a 
study on optimizing the placement of ATOS needs to be initiated. The optimization will 
account for both where ATOS should be implemented, as well as a time phased approach 
to the implementation. These analyses will help decision makers in determining the 
optimal locations that provide the greatest savings due to the large munitions stockpiles. 
For example, small munitions facilities where munitions do not have a high cycle rate 
may not be a high priority for the implementation of ATOS. 

Finally, the RFID tags chosen to be attached to munitions pallets need the 
capability to collect the environmental data that is required by QE personnel to input into 
models to provide Ordnance Managers valuable information about the status of their 


munitions in inventory. 
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I. INTRODUCTION 


A. PURPOSE OF THIS STUDY 

The purpose of this thesis is to analyze the potential cost savings and benefits 
associated with the implementation of an Advanced Technology Ordnance Surveillance 
(ATOS) infrastructure at Navy Munitions Management locations. ATOS will provide 
ordnance managers the ability, on a near real-time basis, to locate accurately and to 
determine continuously the environmental status of high value, low density munitions on 
a near real-time basis while also updating predictions of the future condition and 
performance of the ordnance. In the future, a Navy Munitions Management location will 
be selected to initiate a pilot program to assess and validate the benefits of the 
technology. This study provides a methodology for comparing the cost savings over time 
of having an ATOS infrastructure in place compared to the base case of “as-is” inventory 


management. 
ATOS was chosen for this thesis for two reasons: 


e Importance to the Deputy Under Secretary of Defense (DUSD) due to 
its high visibility in the DOD RFID policy. 


e Example of the process by which a project transitions from an 
Advanced Concept Technology Demonstration (ACTD) in technology 
base to a program of record in the DOD 5000 Acquisition System. 


1. Potential Cost Savings and Benefits 

The promise of an ATOS infrastructure for the ordnance manager is that it can 
provide continuous, comprehensive, and real-time visibility of current inventories. Real- 
time visibility enables ordnance managers and ammunition site managers to avail 


themselves of the following potential benefits: 


e Reduce inventories using more accurate data about supply and demand 


of ordnance. 


e Reduce out-of-stocks based on minimum thresholds 


1 


e Provide a foundation for event-driven optimization to enable the re- 
optimization of the supply chain logistics when a problem occurs, e.g., 


an airplane carrying munitions to the theater breaks down. 


e Reduce manual inventory management tasks, thereby reducing human 
errors and freeing up labor to be reallocated to more mission-critical 


activities 
e Identify and reduce shrinkage across the munitions supply chain 


e Improve efficiencies within munitions warehouses by reducing time 


spent searching for specific items 


e Improve asset management and more accurately validate physical 


inventory with accounting data 


e Improve responsiveness to munitions recalls through greater 
specificity, which will pinpoint exactly which lots of items need to be 


returned and where to find them 


e Maintain or extend munitions shelf life due to timely preventive 


maintenance or calibration 


e Avoid having to replenish stolen or misplaced items since an RFID tag 


can be integrated to an alarm system 


B. WHAT IS RFID? 

Radio Frequency Identification is an Automatic Data Collection (ADC) 
technology that uses radio frequency waves to transfer data between an RFID reader and 
an item of interest to which an RFID device (tag) is attached in order to identify it, 
categorize it, and track it. Compared to the “as-is” inventory management system, RFID 
can greatly reduce the time to conduct an inventory because it is fast, reliable, and does 
not require line of sight or contact between the RFID reader and the RFID tagged item. 


This non-line of sight physical characteristic means that tags can be read through a 


variety of RF propagation media, to include fog, snow, dirt, grime, and other visually and 
environmentally challenging conditions. In these conditions, barcodes or other optically 


read technologies would be useless. 


Present technology allows RFID tags to be read at very high speeds, even in 
adverse weather conditions responding in less than 100 milliseconds [SAVI 2005]. Due 
to its capability and reliability, RFID technology has been adopted in a wide range of 
automated data collection and identification applications that would not be possible 


otherwise. 


1. RFID Characteristics 
There are six important characteristics of RFID that impact the transfer of data 
between a tag and an RFID reader: range, range adjustment, propagation, directionality, 


multi-tag collection, and memory [SAVI 2005]. 


a. Range. Range is defined as the maximum distance for a successful 
communication between the RFID tag and RFID reader. RFID 
technology allows RFID readers to communicate successfully with 
RFID tags in either of the following ranges: 

e Very short range: Up to 60 centimeters (two feet) 

e Short range: Up to five meters (16 feet) 

e Long range: Greater than 100 meters (320 feet) 
Frequency is one of the leading factors that effects range of a 
successful tag-reader communication and determines the type of 
RFID technology that should be used for a_ specific 
implementation. Low frequency systems have short reading range 
and are most commonly used in item tracking and security access 
implementations. High frequency systems offer long read ranges 
and high reading rate and are commonly used for railroad car 
tracking and automated toll collection. Table 1 below displays 


common RFID frequencies being used in RFID readers. 


Frequency Band 
125 — 134 KHz 
13.553 — 13.567 MHz 
400 — 1000 MHz 
2.45 GHz 


Description Range 
Low frequency To 18 inches 


High frequency 3 -10 feet 
Ultra-high frequency (UHF) 10 - 30 feet 
Microwave 10+ feet 





Table 1. Common RFID Frequencies and Passive Ranges (From Ref. Intermec 


Technologies Corporation 2004). 


Range adjustment. Range adjustment is the RFID reader’s ability 
to automatically adjust the range at which it is being operated. Tag 
reader communication is guaranteed within the specified range of 
the reader and tag-reader communication outside the specified 


range is impossible. 


Propagation. Propagation is the ability of the RFID reader to 
perform tag-reader communication through or around objects and 
material. With very good propagation, the RF can penetrate 
through items allowing successful data transfer between the tag 
and reader. Having good propagation enhances the penetration 


through water, liquids or human tissue. 


Directionality. Directionality is the ability of the RFID reader to 
achieve directional RF coverage using directional antennas. There 
are two types of directionality: Omni-directional and Directional. 
Omni-directional coverage has the same RF intensity (or coverage) 
in all directions. With directional coverage, the RF intensity is 


much stronger in one specific direction. 


Multi-tag collection. Multi-tag collection is the ability of the RFID 


reader to process more than one tag within a designated area. 


Memory. Tags range from small memory size (16 bits) to large 
memory (at least 512 kilobytes). Memory is an important 
characteristic in RFID communication because it determines the 
read only (passive RFID), read/write (active RFID), or write-once- 


read-many capabilities in the tag-reader communication. 
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C. HOW DOES AN RFID SYSTEM WORK? 


Figure 1 shows a typical RFID configuration, consisting of 


one or more RFID tags, 

two or more antennas, 

one or more interrogators, 

one or more host computers, and 


the appropriate software to transform the raw data into useable 


information. 


In a basic RFID tag operation, radio waves are used to transfer data between the 


RFID tag (transponder) and the read/write device (interrogator), which is tuned to the 


same frequency. The interrogator sends out a signal, which is received by all tags that are 


present in the radio frequency (RF) field tuned to that frequency. Tags that receive the 


signal respond by transmitting their stored data. The tags can hold many types of data 


about the item, such as its serial number, configuration instructions, what time the item 


traveled through a certain zone, temperature and other data provided by sensors. 
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Figure 1. RFID System Components (From Ref. Intermec Technologies Corporation 


2004). 


The read/write device receives the tag signal, decodes it and transfers the data to 


the host computer system. 


RFID tags can be attached to virtually anything — from an airplane, to a pallet, to a 
case, to an item stored on a shelf. If multiple tags are present in the field of interrogation, 
more efficient RFID implementations have anti-collision algorithms, which determine the 


order of response so that each tag is read once and only once. 


D. ACTIVE RFID TAGS 

The term “active” means that there is an internal battery source installed in the tag 
to provide power to send a signal on demand. Active RFID tags are wireless 
transponders which can automatically identify, locate, track, monitor, and protect a 
variety of items of interest. Unlike Global Positioning System (GPS) which is available 
worldwide, active RFID only operates around the operation site where the RFID 
infrastructure has been implemented. Active tags can be activated for transmitting when 
they reach a particular location, or, they can be programmed to transmit at set intervals, 
or when a change of condition occurs. Changed conditions include movement or sensor 


thresholds such as temperature, humidity, shock or any number of sensor detections. 


Active tags, because of their power source, also have the ability to act without an 


external activation, thereby providing a number of added value functions. For example: 


° A tag can be configured to alarm and send an alert signal if the tag is 
removed, so (anti-tamper) active tags provide a security solution for assets 


and containers. 


° Tags equipped with a motion sensor can alarm in the event of 


unauthorized inventory movement. 


° Beaconing tags can provide an automatic inventory count. Beacon tags 
send signals to a receiver at pre-determined time intervals to provide 


continuous monitoring on the inventory and its location. 


if Active RFID Key Capabilities 

In an August 2004 market study and end user survey by Venture Development 
Corporation [Axcess 2005], active RFID systems, compared to the passive RFID 
inventory management systems (i.e., non-active RFID systems) were found to have a 


series of valuable characteristics, including the following: 


e Enhanced dependability because of high performance, 1.e., active RFID is 
continuously emitting RF energy, thereby, providing the user with the 


latest information 


° Enhanced security/access control including theft reduction 
e Provided the ability to link tags together in software for custodianship 
° Provided the ability to automate identification and location by removing 


human intervention 


° Improved the accuracy and reliability of data 
e Improved read accuracy and longer read ranges than passive RFID 
° Increased data transfer rate 


E. PASSIVE RFID TAGS 

Unlike active tags, passive RFID tags do not have an internal battery source. 
Passive RFID tags rely on capturing and re-using a small portion of the wake-up signal’s 
energy to transmit its RFID tag ID and other information back to the receiver. This 
requires a strong RF signal from a reader and limited available energy constrains the RF 
signal strength returned from the tag. For this reason, passive RFID tags can only operate 
over very short ranges and requires a line of sight for successful operation. This is a 
“good news, bad news” situation [Axcess 2005]. The “good news” is that passive tags 
can be manufactured and sold at much lower prices than active tags because of their 
capability. This can be a critical decision point in many RFID supply chain applications 
requiring literally millions of tags. The “bad news” is that passive tags are not 100% 
reliable because they depend on a small amount of power to push their signal off metal 


surfaces or through layers of palletized items. 
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In short, active tags therefore have an innate performance advantage over passive 
tags when it comes to providing a consistently robust, penetrating signal. When 
activated, active RFID tags send a signal capable of being read at much greater ranges 


than passive RFID tags. 


F. DEPARTMENT OF DEFENSE (DOD) RFID POLICY 

Mr. Michael Wynne, Acting Under Secretary of Defense (USD) (Acquisition, 
Technology and Logistics (AT&L)), announced a new RFID policy on July 30, 2004 
within the DoD [USD 2004]. The DoD requires passive RFID tags on the case, pallet, 
and item packaging for Class I (subsistence and comfort items), Class II (clothing, 
individual equipment, and tools), Class VI (personal demand items), and Class IX 
(weapon systems repair parts and components) commodities delivered on or after January 
1, 2005. For about two years, DoD will accept Electronic Product Code (EPC) Class 0 
(read only) or Class 1 (read/write) passive RFID tags. DoD will migrate to Ultra High 
Frequency (UHF) Generation 2 tags when specification is finalized. In accordance with a 
white paper [Alien Technology Corporation 2005], these Generation 2 tags will be the 
tags of the future. It is expected that Generation 2 tags will have the capability to read 


over 1,000 tags per second. 


The following excerpts were taken from a November 12, 2005 article in New 
York Times, titled “Military To Urge Suppliers To Adopt Radio ID Tags,” reported that 
the Defense Department will begin prodding suppliers to use RFID tags on cartons and 


pallets or goods entering DoD’s vast supply system. 


Further paving the way for a type of inventory tracking technology that 
Wal-Mart is already turning into a commercial standard, the Defense 
Department on Monday will begin prodding suppliers to use radio- 
frequency ID tags on cartons and pallets of goods entering its vast supply 
system. 


The tags, meant to let goods be tracked without the proximity and line of 
sight required by bar-code scanners, have vast potential for military, 
homeland security and commercial applications. But just as Wal-Mart is 
proceeding in steps, starting only with its biggest suppliers, so will the 
military adopt the radio tags in stages. 


The new program will initially apply only to a range of combat-support 
goods handled by the Defense Department's two largest supply centers: 
the Susquehanna depot in New Cumberland, Pa., and the San Joaquin 
depot in Stockton, Calif. And rather than adding the requirement to 
existing contracts, the military will make it a proviso of new or renewed 
contracts. 


But some of the Defense Department's largest suppliers say they will tag 
some goods headed to Susquehanna and San Joaquin even before new 
contracts require them to do so. 


It's a critical part of our effort to support the war fighter and we want to be 
an industry leader, said George Ellis, who oversees radio-frequency ID at 
Raytheon, which has also been testing use of the tags to track goods inside 
its own operations. 


While giants like Raytheon are handling the tagging of military goods 
themselves, many smaller companies are using outside contractors like 
SimplyRFID, a four-year-old consulting and management company based 
in Warrenton, Va. 


"Our traffic is up significantly in the past two weeks," said Carl Brown, 
president of SimplyRFID, which charges customers shipping 5,000 items 
or less to the military roughly $1 a tag. 


Alan Estevez, the Defense Department logistics policy specialist who is 
overseeing the program, emphasized the gradual phase-in of the 
technology. "I'm not expecting a Big Bang on Nov. 14," he said. 


Even so, tens of thousands of the Defense Department's estimated 60,000 
suppliers could come under its requirements within a year, Mr. Estevez 
said. The program calls for adding another 34 supply centers next year and 
the rest of the military's distribution operations in 2007. 


Supporters of the tagging technology said that the military's new rules 
were the biggest step forward for radio tags since Wal-Mart Stores began 
requiring its largest suppliers to use them for shipments to three 
distribution centers last January. Wal-Mart, which also supported the 
tagging at 150 stores served by the distribution centers, has since 
expanded its program to more distribution centers and 500 stores, with 
plans to double that number early next year. 


In theory, the tagging will eventually wring billions of dollars in waste out 
of supply chains, sharply curtail theft and counterfeiting, and reduce the 
frequency of shoppers' encountering empty shelves instead of the products 
they want. The tagging may also speed customers through checkout lanes 
because, unlike bar-code scanners, tag readers can look for numerous 
products with a single signal. 
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In practice, though, figuring out where to place the readers and the tags to 
get data reliably has proved to be an expensive trial-and-error process. 
Compared with the military supply chain, Wal-Mart's distribution system 
is in some ways more difficult to manage than the Defense Department's 
because goods move through the Wal-Mart consumer pipeline so much 
more rapidly. But the military has far more suppliers, along with a 
complex mix of new, replacement and repaired goods, and less predictable 
demands than Wal-Mart's seasonal peaks. 


What is more, "the backroom of a Wal-Mart on a Saturday afternoon may 
be hectic, but it's not like a mobile supply center in the desert that's being 
shot at," Mr. Estevez said. The military's backing gives other users and 
investors confidence in radio tags, said Kevin Ashton, vice president of 
ThingMagic, a manufacturer of tag-reading devices based in Cambridge, 
Mass. "You have to give people the sense that this is inevitable to have the 
opportunity to work out the details," he said. 


The details have been a challenge. The Defense Department and private- 
sector trucking and shipping companies have plenty of experience with 
battery-powered tags able to communicate with satellites to track large 
containers and expensive equipment. 


The microchips in the tags now being introduced by the military and Wal- 
Mart, however, rely on power emitted from the scanners to provide the 
energy the tags need to respond. Plans call for applying these "passive" 
tags to billions of items before the end of the decade. 


Researchers at the University of Arkansas reported last month that 12 
Wal-Mart stores with radio tag scanners in their storerooms were able to 
restock shelves at three times the speed of a control group of 12 stores that 
relied on traditional methods to locate stock. The tag stores were 16 
percent less likely to have empty shelves. 


Wal-Mart hailed the study as proof of the technology's value. But it did 
not quantify the actual savings. And AMR Research, a consulting 
company, concluded from a survey of major Wal-Mart suppliers that it 
would take them more than nine years to earn a payback on their 
investments at current tag and reader prices - far beyond the one-to-two- 
year time horizon that would make the technology financially attractive. 


The military may encounter far less resistance. Unlike Wal-Mart, the 
government expects to shoulder the extra costs the technology imposes on 
suppliers. And the military's goals seem less likely to stir up opposition 
from privacy advocates who are opposing use of the tags in commerce. 


"The real payback for D.O.D. is to have the soldier on the ground have the 
part needed to make a tank run or a plane fly," said Mr. Estevez. 
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G. PREVIOUS RFID WORK DONE AT THE NAVAL POSTGRADUATE 
SCHOOL (NPS) 


There have been four Master of Business Administration (MBA) professional 
reports completed at NPS that researched different approaches to the employment of 
RFID within the United States military. 


The first two projects considered the impact of active RFID in the supply logistics 
chain. There is a current focus on supply logistics chain activities, as part of several DoD 
initiatives to provide total product visibility of all cargo movements in support of the 


troops. 


The third study conducted a cost-benefit analysis of implementing an active RFID 


infrastructure as a real-time asset management tool within a military hospital. 


The fourth thesis conducted an analysis of cost and benefits associated with the 
implementation of an active RFID infrastructure at a maintenance depot. These theses 


are summarized below. 


1. USTRANSCOM and In-transit Visibility 

This NPS thesis was completed in December 2003 [Hozven 2003] and focused on 
the Air Mobility Command (AMC), an organization under the United States 
Transportation Command (USTRANSCOM) which is accountable for all military 
transportation and is responsible for establishing the global RFID infrastructure. The 
primary objective of this study was to assess the potential value of RFID to AMC if used 
in its worldwide network of ports to manage the supply logistics chain. The study 
concludes that there are benefits to using RFID but more pilot projects are needed to be 
implemented to fully quantify the value of RFID technology. This thesis in PDF format 


can be accessed at the following link: 


http://stinet.dtic.mil/cgi-bin/GetTRDoc? AD=ADA420561 &Location=U2&doc=GetTRDoc.pdf 





Zs Value of Supply Chain Logistics Information 
This NPS thesis was completed in June 2004 [Corrigan 2004] and attempts to 
quantify the value added by using RFID. It does this by investigating what a Supply 
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Officer was willing to pay for real-time information and visibility of products in the 
supply logistics chain. The report concludes that real-time logistics information is 
valuable in order to effectively support the warfighter. Additionally, the thesis noted that 
RFID offers tangible cost savings to include reduction in labor costs, as well as intangible 
benefits of better access to information to allow for better management decisions 
concerning allocation of limited resources. This thesis can be accessed in PDF format at 


the following link: 


http://stinet.dtic.mil/cgi-bin/GetTRDoc? AD=ADA424676&Location=U2&doc=GetTRDoc.pdf 





3. Management of Medical Equipment 
This NPS thesis was completed in December 2004 [Sanchez 2004], and its 


primary focus was to identify the value of RFID in the management of medical 
equipment at the Naval Medical Center in San Diego, California. The study concluded 
that the value of RFID was two fold: 


e Cost savings produced by the elimination of replacement costs caused by 


lost equipment 


e Increased efficiency in manpower utilization by avoiding the time required 


to find lost equipment. 


The study presented a Return on Investment (ROI) analysis that illustrated a 
positive Net Present Value (NPV) in less than a year when RFID was implemented under 


the study conditions. This thesis can be accessed in PDF format at the following link: 


http://stinet.dtic.mil/cgi-bin/GetTRDoc? AD=ADA429394&Location=U2&doc=GetTRDoc.pdf 





4. Management of Equipment at Tobyhanna Army Maintenance Depot 

This NPS thesis was completed in June 2005 [Miertschin 2005] and its purpose 
was to identify the potential value of RFID used for inventory and asset management at 
the Tobyhanna Army Maintenance Depot. This study concludes that RFID 
implementation proved beneficial to increase process efficiency and reduce the number 
of man-hours expended to find lost or misplaced equipment. This study “indicates a ROI 


of less than one year. This result supports other research conducted on RFID as an asset 
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management tool, and indicates that Tobyhanna’s investment in advancing technology 
essentially paid for itself within one year when measured in labor cost savings. More 
importantly, the technology yields an annual savings of 837 RCT days. Although we 
could theoretically assign a monetary value to the RCT savings, we believe the statistic 
adequately reflects the savings to the field.” This thesis can be accessed in PDF format at 


the following link: 
http://library.nps.navy.mil/uhtbin/hyperion/05Jun%5FMiertschin%5FMBA.pdf 
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Il. BACKGROUND 


A. ADVANCED CONCEPT TECHNOLOGY DEMONSTRATION PROCESS 
1. Introduction 
Budget constraints, significant changes in threats, and an accelerated pace of 
technology development have challenged the ability of the Component Commanders 
(COCOMs) to adequately respond rapidly to the evolving military needs. Part of the 
Department of Defense response to thee challenges has been to initiate the Advanced 
Concept Technology Demonstration program in early 1994 to get new technologies into 


the hands of the warfighter as quickly as possible. 


The ACTD program is designed to assist the DoD acquisition process adapt to 
today’s economic and threat environments. ACTDs identify significant military needs 
and match them to mature technologies or technology demonstration programs which are 
maturing key technologies in order to solve important military needs (see Figure 2). 
These technologies are then combined and integrated into a complete military capability 
to provide decision makers an opportunity to understand fully the operational potential 
offered by a proposed new military capability before making an acquisition or 
sustainment decision. This goal is met by developing fieldable prototypes of the 
proposed capability and providing those prototypes to the warfighter for evaluation of 
that capability. The warfighter evaluates the capability in real military exercises and at a 
scale sufficient to assess fully military utility. During the ACTD, the warfighter also 
evolves the broad statement of need, which existed at the start of the ACTD, into a 
definitive set of operational requirements that can support a follow-on acquisition. At the 
completion of the ACTD, the prototypes used in the evaluation process are left with the 
warfighter to provide an interim capability or, in some cases, to fulfill the total, current 


need. 


Figure 2 below displays the ACTD Development Process. 
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Figure 2. ACTD Development Process (From Ref. ACTD 2004). 


In a February 24, 2005 article in Inside the Pentagon, titled “DOD Plans New 
Acquisition Executive Post To Champion Joint Programs,” plans were announced on 


establishing the policies for ACTDs and JCTDs. 


The Pentagon plans to establish a new acquisition executive to champion 
technologies and concepts designed for joint operations, according to 
defense officials and documents. 


The creation of the new position is part of a wider effort to overhaul the 
advanced concept technology demonstration program -- the Pentagon's 
marquee project for rapidly fielding new technologies -- into another effort 
called the joint capability technology demonstration program. 


The goal is to expedite deliveries of new technologies to soldiers, sailors, 
airmen and Marines by putting in place new funding mechanisms and 
organizations to make sure new, proven weapons and combat technologies 
are designed for use by more than one service and not orphaned by 
individual services at budget time. 


Beginning Oct. 1, the Pentagon plans to have in place the new acquisition 


executive to ensure "cradle to grave" funding and advocacy for promising 
technologies that do not have clear champions in the Army, Navy, Air 
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Force or Marines. This new post would be equal in rank to the service 
acquisition executives, according to Defense Department officials. 


To facilitate this undertaking the Office of the Secretary of Defense has 
shifted $40 million in its fiscal year 2006 budget proposal from the ACTD 
budget line to initiate JCTD programs. 


"This is just seed money," said Mark Peterson, head of program resources 
and integration for the deputy under secretary of defense for advanced 
systems and concepts, in a Feb. 22 interview. "We expect in next year's 
budget that this might change." 


So do senior Pentagon officials. In a late December budget decision, 
Deputy Defense Secretary Paul Wolfowitz directed the under secretary of 
defense for acquisition, technology and logistics and the chairman of the 
Joint Chiefs of Staff to establish an improved process to transition 
promising ACTDs to acquisition programs and bring forward a new 
spending proposal this summer for consideration in the FY-07 budget. 


Pentagon officials say the joint capability technology demonstration effort 
is designed to deliver the improved transition process Wolfowitz seeks. It 
is the brainchild of Sue Payton, the deputy under secretary of defense for 
advanced systems and concepts, who has worked for the last 18 months to 
improve the ACTD process and put in place what she calls a new business 
model for rapidly fielding new technologies desired by combatant 
commanders. 


Beginning in 1994, the ACTD program established an alternate route to 
quickly put new technologies in the hands of warfighters. The program 
takes new but relatively mature technology and offers the services the 
opportunity to assess prototypes in a military environment. Targeted to 
address pressing requirements, ACTDs typically spend three to four years 
in development, after which a handful of prototypes are delivered to 
military units. They spend as many as two more years evaluating the 
technology for operational usefulness. 


This four- to six-year cycle is faster than the traditional acquisition cycle, 
which can take between 10 and 15 years from the concept stage to 
fielding. 


In some cases, technologies developed through ACTDs are used primarily 
by a single service. At the end of the demonstration, the service can buy 
more of the capability or walk away from the project. Many ACTDs, 
however, are designed expressly for commanders who are seeking to 
improve the coordination and operations of service-specific technologies 
that weren't designed to work together. In some cases, the objective is to 
acquire a technology none of the services provides. 
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Once an ACTD is complete, the four-star combatant commanders who 
sponsor them must depend on one of the services to acquire the 
technology and fund its operational use, although U.S. Special Operations 
Command has unique acquisition authority and is exempt from this rule. 
The services, however, do not always rank ACTDs desired by combatant 
commanders high in their procurement portfolios. 


"Military services and defense agencies have been reluctant to fund 
acquisition of ACTD-proven technologies, especially those focusing on 
joint requirements, because of competing priorities," the Government 
Accountability Office said in a December 2002 report. 


The Pentagon's FY-06 budget request includes $40 million to kick start a 
number of proposals aimed at correcting key difficulties that have surfaced 
in guiding new technologies from government and commercial 
laboratories to troops and into the Pentagon's acquisition and operations 
accounts. 


These funds will be spread across four new program elements to fund 
JCTDs as well as a pilot program to establish a new defense acquisition 
executive. 


This new position would share rank with the service acquisition executives 
and be the primary advocate in the budget process for joint capabilities 
that do not have a natural place in any of the service accounts. 


Key to the new approach is a change in how projects are funded. In order 
to remain as responsive to the current needs of combatant commanders, 
the Pentagon keeps ACTDs out of its planning, programming and budget 
execution cycle, which involves a two-year delay between requesting and 
receiving funds. 


"So every time an ACTD starts, if a service has not already been planning, 
you have to break [another] program" to find the money for the new 
project, said Peterson. The net effect: "We create an instant unfunded 
requirement," he said. 


Under the JCTD approach, the Office of the Secretary of Defense will 
provide more funds at the beginning of a project, boosting its start-up 
contribution from 30 percent to at least 50 percent in order to reduce the 
pressure on the services to find money for the project outside of the budget 
cycle. 


The JCTD process will set shorter time lines for demonstrating new 
concepts or technologies. 


"We would like to make that quicker by at least a year," said Peterson. 
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The JCTD strategy will requires a final demonstration in two to three 
years, faster than the three-year to four-year goals for most ACTDs. In the 
first year, JCTD officials will be required to deliver a preliminary 
capability. 


After that, they must be 50 percent complete by the end of the second year 
and wrap up in the third year. Payton also wants 80 percent of JCTDs to 
transition at least half of their products into a permanent place in the 
Pentagon's budget. 


The following are some examples of FY06 ACTDs and JCTDs. These 
ACTD/JCTDs have the potential to support COCOM missions, whether or not they 
evolve into full-fledged programs of record. For a full list of ACTDs and JCTDs, refer to 


Appendix A and the following web link. 


http://www.acq.osd.mil/actd/descript.htm 


Comprehensive Maritime Awareness (CMA) JCTD 


CHAMPION (Counter Intelligence-Human Intelligence Advanced 
Modernization Program/Intelligence Operations Now) JCTD 


Extended Space Sensors Architecture (ESSA) 

Joint Modular Intermodal Distribution System (JMIDS) JCTD 

Large Data JCTD 

Multi-service Advanced Sensors to Counter Obscured Targets (MASCOT) 
Joint Enable Theater Access (JETA) 

Event Management Framework (EMF) 

Node Management And Deployable Depot (NOMADD) 


Small UAV 


At the conclusion of the ACTD operational demonstration, there are three 


possible outcomes. 


e First, recommend acquisition of the technology. 
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e Second, if the capability or system does not demonstrate military 


utility, the project is terminated or returned to the technology base. 


e Third, the warfighter’s need is fully satisfied by the fielded prototype 
capability that remained onboard and there is no need to acquire 


additional units. 


2. ACTD/JCTD Transition Process 
Figure 3 below outlines possible paths which the ACTD in review might follow 


as it transitions to a program of record. 
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Figure 3. Alternatives Following Completion of ACTD (From Ref. ACTD 2004). 


Transition to the formal Defense acquisition process will be necessary when 
development or production is required. The acquisition category will depend on both the 
number and cost of systems required to meet the military need. The next step is to 
determine at what point does the ACTD enters the acquisition process. If significantly 
more development of the technology is required, the system might enter into the 
development portion of the Engineering and Manufacturing Development (EMD) phase. 
On the other hand, if the capability of the ACTD is sufficient and needed promptly, 
entering into the Low-Rate Initial Production (LRIP) portion of EMD is an option. 
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There are three generic classes of ACTDs that present significantly different 


transition challenges (see Figure 4): 


a. Class I ACTD. These are typically informational systems with special 
purpose software operating on commercial workstations. They frequently are required in 
small quantities, and that requirement can be satisfied without further development or 
production using the residual ACTD system (residual ACTD systems are the systems 
used during the ACTD that are left behind with the warfighter to meet his military need) 
or a few additional systems [ACTD 2004]. 


b. Class If ACTDs. These are weapon or sensor systems similar in concept 
to systems that are acquired through the formal acquisition process. In some cases a 
Class If ACTD will be planned ahead of time to transition into LRIP following ACTD, 
but at other times it is appropriate to plan for additional development following the 


ACTD [ACTD 2004]. 


G Class III ACTDs. These ACTDs are best described as “systems of 
systems.” This means that an individual element within the overall system of a Class HI 
ACTD may be a fielded system, a system already in acquisition, or a system emerging 
from the technology base. The overall ACTD may involve multiple Program Executive 
Officers (PEO), and perhaps multiple Military Departments. The challenge here is to 
integrate and coordinate the individual transitions to achieve the capability presented in 


the ACTD [ACTD 2004]. 
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Figure 4. Classes of ACTDs (From Ref. ACTD 2004). 


B. ADVANCED TECHNOLOGY ORDNANCE SURVEILLANCE (ATOS) 


1. Introduction 

From major distributions sites, such as, Naval Surface Warfare Centereapons, 
Indian Head Division, Maryland, and Crane Army Ammunition Activity, Crane, Indiana, 
to the warfighter, the effective management practice of overseeing munitions has long 
been hindered by labor-intensive inventory methods and stockpile inaccuracies. The 
status quo (or “as-is”) transaction process involves tedious manual data entry using 
inventory sheets to track and monitor the flow of munitions. The loss of asset visibility 
due to database munitions inaccuracies adds to the creation of a host of problems for the 
acquisition manager, logistician, storage custodian, and most importantly, the warfighter 
in the field. These problems cause the overall mission readiness to be degraded, because 
needed munitions either cannot be located, or when located, their reliability is unknown 
because the current surveillance methods fail to provide the critical environmental 


information required to determine munitions’ serviceability and reliability. Knowing 
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munitions storage conditions, specifically temperature, relative humidity, and gravity 
shock, can have significant impact on munitions’ safety and reliability. By taking 
advantage of the latest in Automatic Identification Technologies (AIT) and Micro- 
Electromechanical Systems (MEMS), all stakeholders should have a capability that 
provides near real-time environmental information and automated support for managing 


their respective munitions stockpiles. 


Micro-Electrical-Mechanical Systems (MEMS) are promising new technologies 
that facilitate a new dimension of Focused Logistics support. As the website 


http://www.memsnet.org/mems/what-is.html describes it, 





MEMS is the integration of mechanical elements, sensors, actuators, and 
electronics on a common silicon substrate through microfabrication 
technology. While the electronics are fabricated using integrated circuit 
(IC) process sequences (e.g., CMOS, Bipolar, or BICMOS processes), the 
micromechanical components are _ fabricated using compatible 
"micromachining" processes that selectively etch away parts of the silicon 
wafer or add new structural layers to form the mechanical and 
electromechanical devices. MEMS promises to revolutionize nearly every 
product category by bringing together silicon-based microelectronics with 
micromachining technology, making possible the realization of complete 
systems-on-a-chip. MEMS is an enabling technology allowing the 
development of smart products, augmenting the computational ability of 
microelectronics with the perception and control capabilities of 
microsensors and microactuators and expanding the space of possible 
designs and applications. 


Microelectronic integrated circuits can be thought of as the "brains" of a 
system and MEMS augments this decision-making capability with "eyes" 
and "arms", to allow microsystems to sense and control the environment. 
Sensors gather information from the environment through measuring 
mechanical, thermal, biological, chemical, optical, and magnetic 
phenomena. The electronics then process the information derived from the 
sensors and through some decision making capability direct the actuators 
to respond by moving, positioning, regulating, pumping, and filtering, 
thereby controlling the environment for some desired outcome or purpose. 
Because MEMS devices are manufactured using batch fabrication 
techniques similar to those used for integrated circuits, unprecedented 
levels of functionality, reliability, and sophistication can be placed on a 
small silicon chip at a relatively low cost. 


Through the use of real-time sensor data, MEMS provides improved situational 


awareness and asset visibility within a small and inexpensive form. MEMS technology 
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uses modern electronics fabrication techniques with micro machining to provide 
advanced functionality on a silicon chip. Through proof-of-concept projects, U.S. Army 
Logistics Transformation Agency (LTA), a field operating agency of the Deputy Chief of 
Staff, G-4, is bringing active Radio Frequency Identification (RFID) devices together 
with MEMS to capture, record, and communicate temperature, humidity, shock, light and 
other environmental conditions. When combined with track and trace capability of active 
RFID, these prototype devices can provide stand-off asset visibility, self-reporting 
communications, and data storage functions to record, alert and provide immediate 
feedback to soldiers and logisticians on the condition of assets. Potential applications for 
MEMS span all classes of supplies and logistics processes including perishable 
subsistence, diagnostics/prognostics in maintenance, individual protective equipment, 
medical supplies, ammunition, fuel, inventory and shelf life management, and weapon 


system condition “health” monitoring. 


The LTA is examining the application of commercially available MEMS-based 
sensors with active RFID to track and report environmental factors that affect the 
viability of Class VIII medical supplies. At present, hundreds of thousands of dollars are 
expended annually on the cost of labor to accomplish manual processes related to 
monthly visual monitoring of humidity control devices on Class VIII DEPMEDS in long 
term storage at Sierra Army Depot (SIAD). Additionally, millions of dollars of Medical 
Sets, Kits and Outfits (MESKOs) materiel had to be destroyed by U.S. Army Medical 
Materiel Center-Europe (USAMMC-E) in 2003 due to the effects of unknown 
environmental conditions that the materiel had been exposed to while in transit or 
storage. LTA has developed a high level RFID/MEMS conceptual design analysis; a 
Business Process Analysis (BPA) for MEMS application to Class VHI addressing high- 
level analyses of Deployable Medical Systems (DEPMEDs), Medical Sets, Kits and 
Outfits, and Military Vaccines; and has tested RFID MEMS technologies in a laboratory 


setting. 
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2 System Description and Capabilities 
ATOS is an RFID-based automated system that provides a capability to collect 
environmental data in near real-time and supports munitions management tasks on stored 


munitions pallets [AFOTEC 2004]. 


The implementation of ATOS data rich RFID tags is not meant to replace any of 
the traditional business processes of Ordnance Management (OM). Rather, ATOS tags 


are meant to enhance many current processes and add additional capabilities to OM. 


The following are six ATOS oriented munitions management tasks which form 


the foundation for inventory management of munitions: 
e Munitions Receipt 
e Inventory Maintenance 
e Munitions Movement 
e Munitions Issue 
e Munitions Transfer 
e Munitions Quality Assurance (QA) 


Figure 5 depicts graphically how ATOS automates these tasks 
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ATOS Automated Munitions Management Functions (From Ref. 
AFOTEC 2004). 


The ATOS system consists of six major components: 


System (AIS). 


1. 
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RF tag, 

Handheld Reader (HHR), 

Reader Control Unit (RCU), 
Pre-Processor (PP), 

Environmental Database (EDB), and 


An interface to service ammunition Automated Information 
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Radio Frequency Extenders (RFE) are used to enhance RF coverage inside larger 


enclosed magazines or depots. 


The RF tag is a small enclosure attached to a munitions pallet using an RF tag 
holder. The RF tag contains temperature, relative humidity (RH), and gravity-shock 
sensors. The temperature and RH sensors collect environmental data according to a 
specified period of time. The RF tag also has a receiver, transmitter, and non-volatile 
memory to receive, transmit, and store unique asset information and environmental data. 
The HHR is a portable wireless barcode scanning device with the capability to read and 
write asset data to and from the RF tags and reads linear and two-dimensional barcodes. 
Using a keypad and touch screen, the user can upload and download asset information to 
and from the RF tags using a function called roll call, which queries the RF tag to identify 
itself and provide any changes to asset and environmental data. Data are stored in the 
HHR as transaction records until they are transferred to the PP for analysis and archived. 
The HHR can also retrieve stored RF tag data from the RCU using an Ethernet cable. The 
HHR uses a Windows Pocket Personal Computer operating system, which is a 


commercial off-the-shelf technology (COTS) software. 


The RCU is a fixed RF reader that collects asset and environmental data from the 
RF tags. The RCU is powered by 24 volts direct current (VDC) and can be mounted 
inside a munitions magazine. Depending on the size of the magazine, RFEs are connected 
in series with the RCU to achieve optimal RF transmission coverage. The RCU obtains 
data from the RF tags using a function called interrogation, which consists of two 
independent reads (short and long). The short read determines the number of RF tags 
detected, RF tag identification (ID), Department of Defense Identification Code 
(DODIC), National Stock Number (NSN), consignee, any environmental sensor flags, 
and a low battery flag. The time duration for a short read depends upon the number of 
RFEs connected in series with the RCU, the number of RF tags detected inside the 
storage facility, and the type of material (wooden crates and aluminum pallets) on which 
the RF tags were attached. The long read downloads the remaining asset information and 
environmental data. The duration of a long-read can take hours to days depending on the 
number of RF tags being interrogated. The RCU serves as a long-term data storage unit 


for receiving and storing interrogated RF tag data. The RCU stores these data until the PP 
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commands it to transfer the data via a wireless local area network (WLAN), local area 
network (LAN), serial cable connection with the HHR. The RCU has no direct user 


interface, i.e., no keyboard or display. 


The PP is an interactive command and control system designed to retrieve RF tag 
data from the RCU and HHR. The PP is typically located in the central work area. 
Munitions information is formatted and passed from the PP to the service ammunition 
AIS, such as the Retail Ordnance Logistics Management System for the U.S. Navy local 
and theater inventory management of munitions, and the EDB. Munitions experts and 
analysts can use the Environmental Database to evaluate munitions performance 


fluctuations due to environmental changes. 


Figure 6 shows the connectivity for the ATOS system components. 
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Figure 6. ATOS System Architecture (From Ref. AFOTEC 2004). 
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Hil. ATOS BUSINESS CASE ANALYSIS 


A. WHAT IS A BUSINESS CASE ANALYSIS (BCA)? 

A Business Case Analysis examines and compares the benefits, costs, and 
uncertainties of each alternative to determine the most cost effective means of meeting 
the objective. It is a systematic approach to the problem of resource allocation, 


comparing two or more alternatives in terms of cost and benefits. 
The standard steps in a BCA are below: 
e Objectives of the action being considered 
e Specification of assumptions/constraints 
e Identification of alternatives 
e Listing of benefits for all feasible alternatives 
e Cost estimates for each feasible alternative 
e A ranking of alternatives in terms of costs and benefits 
e  Risk/uncertainty analysis 
e Conclusions/recommendations 


Decisions about mission and business planning are important because they 
become the basis for deciding whether an acquisition will be funded. Budget constraints 
are the main reasons that force organizations to justify their spending, because if there 
were no budget constraints, then we could buy anything and everything, and not have to 
engage in making choices. A BCA is a process for preparing a structured proposal that 
establishes sound business decisions for proceeding with an investment/project by 
providing decision makers with the insight into how the investment/project supports the 
business needs and strategic goals. The BCA structures the assessment by providing 
necessary information concerning the scope, alternatives considered, estimated costs and 
Return on Investment (ROI), and risks necessary for decision makers to make an 
informed funding decision for the investment/project. Therefore, budget constraints 


cause an increase in the usage of Business Case Analysis (BCA). 
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Each BCA will be different depending on its application. However, a BCA 


structure should include the following as a minimum [ACC 2004]. 


Introduction. It presents the objectives addressed by the subject of the 
case, and all the options, including the status quo, considered to 


achieve the objective. 


Assumptions and Methods. Outlines the rules for deciding what 
belongs in the case, and what does not, along with the critical 


assumptions. 
Business Impacts. The main business case results. 


Sensitivity and Risk Analysis. Shows how results depend on the 
important assumptions (“what if”), as well as the likelihood for other 


results to surface. 


Conclusions and Recommendations. Recommends specific actions 


based on business objectives and the results of the analysis. 


1. The BCA Process 
Figure 7 displays the BCA process consisting of four steps. 


Definition is the first step in the BCA process and sets the scope of the 
problem. During this step, the assumptions and the constraints are 
formulated which will guide the analysts of the BCA Team throughout 
the process. Also in this step, alternatives to be considered are 


identified, as well as the measures. 


Data Collection is the second step in the BCA process and is where the 
source and types of data to be collected are identified. The collection 
of data may be difficult because the data may be obscured in databases 


in remote locations or buried in budget documents. 


Evaluation Analysis is the third step in the BCA process and is where 


the “number crunching” is done. In this step, the data that was 
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collected in the second step is used to build a case for each alternative, 
using both qualitative and quantitative data. Each alternative is 
compared against each other, in an effort to identify a best alternative. 
It is important that analysts should not only seek to determine which 
alternative has the lowest cost, but which alternatives provide the 


optimal combination of price and performance. 


Results Presentation is the fourth step in the BCA process. This is a 
critical step because if the BCA Team is unable to communicate 
effectively the results to the decision makers, the analysis is worthless. 
Conclusions are to be organized around the objectives stated up front 
in the case. The recommendation of the BCA will consist of 
recommending to stay with the status quo or to adopt the alternative(s) 


being considered. 


Figure 7. 





The Business Case Analysis Process (From Ref. ACC 2004). 
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B. WHAT IS A WORK BREAKDOWN STRUCTURE (WBS)? 

A Work Breakdown Structure (WBS) is a hierarchical approach to plan and 
integrate the various parts of a project. The Department of Defense Handbook, MIL- 
HDBK-881, defines a WBS as follows: 


e <A product-oriented family tree composed of hardware, software, 


services, data, and facilities. 


e The family tree results from systems engineering efforts during the 


acquisition of a defense materiel item. 


The following are a few benefits of how a WBS may assist a Program Manager 


(PM) during the life cycle of a program [MIL-HDBK-881]. 


e Separates a defense materiel item into its component parts, making the 
relationships of the parts clear and the relationship of the tasks to be 


completed-to each other and to the end product-clear. 


e Affects significantly planning and the assignment of management and 


technical responsibilities. 


e Assists in tracking the status of engineering efforts, resource 
allocations, cost estimates, expenditures, and cost and _ technical 


performance. 


e Helps ensure that contractors are not unnecessarily constrained in 


meeting item requirements. 


By displaying and defining the efforts to be accomplished, the WBS becomes a 
management blueprint for the product and reduces the likelihood of something falling 


through the cracks. 


1. Work Breakdown Structure for ATOS 
To be able to provide a realistic BCA of the implementation of an ATOS 
infrastructure at a Navy Munitions Management location, the advice of Subject Matter 


Experts (SMEs) from Naval Surface Warfare Center (NSWC), Indian Head Division, was 
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seek to validate the additional sustained cost in fielding ATOS. The top level WBS is 


composed of the following components: 
e Investment 
e Operation and Support (O&S) 
e Environmental 


The ATOS infrastructure at a Munitions Management facility will consume the 
biggest portion of the investment funding due to the amount of equipment/software, 
installation cost, and personnel training needed for the newly fielded system. Of course, 
the number of munitions magazines and the number of munitions in a facility will dictate 


the true implementation cost. 


Operation and Support consists of the costs needed to sustain the ATOS 
infrastructure. The recurring costs are the cost incurred to continue the validation of 
models and to maintain the readiness of the system, i.e., to replace equipment/software 
due to damage, normal wear and tear, and system upgrades throughout the life cycle of 


the system. 


Critical environmental information, e.g., munitions storage conditions, is needed 
to determine munitions’ safety, reliability, and performance. The status of the munitions 


is critical for the following three reasons: 


1. Safety. Munitions accidentally exploding in munitions magazines 


or afloat without warning pose great danger to personnel and property. 


2. Reliability. Warfighter should have reliable munitions they 
depend on a daily basis. When munitions is intentionally launched or fired (pulled the 
trigger), the munitions should leave the canister of a Vertical Launch System (VLS) or 


the muzzle of a gun without delay. The life of the warfighter may be on jeopardy. 


2. Performance. When the munitions have been launched, they are 
expected to perform to their specifications, i.e., fly at their intended speed and correct 


altitude, for mission success and accomplishment. 
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The collection of environmental data, such as time-phased histories of 
temperature, humidity, and shock for the munitions under surveillance, is for munitions 
incident prevention. Knowing the reliability and the shelf life of munitions are key in 
munitions incident prevention. Being conscious of the storage/environmental factors 
munitions are experiencing or have experienced in the field, can lead to the prevention of 
munitions magazines from inadvertently exploding due to munitions instability or shelf 
life expiration, thereby, saving in the clean-up effort, collateral damage to property, or 


more importantly, the safety of personnel. 


We spoke to Subject Matter Experts whose recollection is that incidents have 
occurred only ashore, and not afloat. This is because ashore munitions management 
facilities intentionally accelerate (“cook-off”’) the shelf life of munitions. This serves as a 
munitions plan to advise afloat units about the serviceability and reliability of their 


munitions onboard. 


Subject Matter Experts also say that two incidents, one in 1970 and one in 1994, 
are the only incidents that have occurred in the last three decades, and that in these two 
incidents, gun propellant was the common denominator. That is about one incident every 


twelve years or 0.1 incidents per year. 


ATOS will play a key role in Quality Evaluation (QE), also known as 
Surveillance. The mission of QE is to determine the safety, reliability, and performance 
of munitions. When QE is accomplished, munitions are dissected to determine the status 
of similar munitions. Obviously, dissected munitions are lost munitions to the inventory. 
According to Subject Matter Expert, Mr. Roger Swanson, NOSSA (N8), Director 
Weapons Assessment, it is impossible to conduct QE on all DoD’s munitions stockpiles 
because of funding constraints and time. For this reason, the current program only has 
QE personnel conduct QE on selected munitions that have the greatest susceptibility to 
the environment. For example, QE on a missile’s rocket motor is accomplished once 
every three years depending on the rocket’s propellant composition. Under the current 
practice, munitions are added to the QE list only when a mishap occurs. This procedure 


has the disadvantage that we react after the fact, i.e., after a mishap has occurred. ATOS, 
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with its potential to collect the necessary environmental munitions data fixes this 


disadvantage because QE will be done on all munitions vice just the selected few. 


The RFID tags attached to munitions pallets will have the capability to collect 
data on temperature, humidity, and shock of all munitions. These data, provided to QE 
personnel, can be analyzed with the right models to predict with a high degree of 
accuracy the safety, reliability, and the performance of the munitions the warfighters are 
employing in theater on a daily basis. The funding for model development and 


implementation will be a recurring cost for fielding ATOS. 


c. “AS-IS” COST (A WORLD WITHOUT ATOS) 


1. Ordnance Management Process 

The current Ordnance Management process is labor intensive. It entails the 
manual data entry using inventory sheets to track and monitor the Department of Defense 
munitions stockpiles. In this process, personnel physically have to conduct every 
munitions management task, i.e., physically count all munitions in a munitions magazine 
when conducting an inventory. Many man-hours are spent doing any one task which can 
be translated to hundred of thousands of dollars per year. Additionally, the manual effort 


is subject to errors, which then yield an inaccurate database. 


The following six munitions management tasks that form the foundation for the 
inventory management of munitions used in the Military Utility Assessment by AFOTEC 


demonstrates how labor intensive the current process can be [AFOTEC 2004]. 


1. Munitions Receipt. When munitions arrive at a Munitions 
Management facility, personnel need to be available for receiving the munitions, move 


the munitions to the storage location, and add the munitions to the current inventory. 


2. Inventory Maintenance. Personnel physically need to count all 
munitions in a munitions magazine to maintain inventory information on all magazines. 


The inaccuracy of this inventory information is subject to human error. 


3. Munitions Movement. When munitions are moved from one 


magazine to another, personnel need to do the manual data entry that reflects the change 
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to the magazine inventory. Again, the inaccuracy of this inventory information is subject 


to human error. 


4. Munitions Issue. When munitions are issued to the warfighter, 
personnel need to update the magazine inventory and the munitions database to reflect 
the current inventory. Again, the inaccuracy of this inventory information is subject to 


human error. 


5: Munitions Transfer. When large amounts of munitions are shipped 
to another Munitions Management facility or to the warfighter in theater, personnel need 
to manually conduct the data entry to reflect the munitions transfer at both the munitions 
database and the magazine inventories. Again, the inaccuracy of this inventory 


information is subject to human error. 


6. Munitions Quality Evaluation/Surveillance. When QE personnel 
remove munitions for inspection, the results of the inspection or condition of the 
munitions need to be recorded in the munitions logbooks. Again, the inaccuracy of this 


inventory information is subject to human error. 


2; Ordnance Management Cost 

The current Ordnance Management process used to manage DoD’s large 
munitions stockpiles does not track the status of all munitions in the inventory, but yet is 
costing DoD millions of dollars a year. The following describe why the current 


management process of munitions has caught the attention of DoD. 


Demilitarization of all munitions is performed and paid by the U.S. Army. The 
Department of the Navy alone in FY00 generated over 16,000 tons of ordnance that 
needed demilitarization and disposal at an average cost of $892 per ton of material for a 
total cost of over $14M. The Army’s disposal budget would be greatly reduced if the 
services found a way to expend a greater portion of its munitions before they become 


obsolete and require high cost to dispose [Herb]. 


Corrosion is another challenge for DoD. It affects mission readiness and adds 


additional cost in shipping replacement munitions. Corrosion is defined as a chemical 
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reaction between a material, usually a metal, and its environment that produces a 
deterioration of the material and its properties. Changes in the environment, 1.e., changes 
in temperature and humidity, cause munitions to corrode thereby costing DoD millions of 
dollars in corrosion maintenance. While corrosion is a problem for all services, not just 
the USAF, we can gain an appreciation of the magnitude of the problem reported in the 
final report submitted February 16, 2005, by C* Technology, Inc, to the USAF on the cost 
of corrosion. This report states that the USAF in FY04 spent $43M in munitions 


corrosion maintenance. 


Munitions handling can be a dangerous evolution because munitions can be 
accidentally dropped and depending on the condition of the munitions, they can either 
explode or they do not explode. If munitions are dropped and they do not explode, the 
current procedure has these munitions shipped to a location where they can be 
investigated for damages. This would be great for single munitions, e.g., a missile. Ifa 
pallet of munitions is dropped instead, the entire pallet is being shipped to be checked for 
damages because personnel cannot determine the condition of the munitions in the pallet. 
Not all munitions may be damaged, but there is currently no way of determining the 
condition of all munitions in the pallet. The inability to determine the condition of the 
munitions affects mission readiness and adds additional cost in shipping replacement 


munitions. 


The purpose of the Quality Evaluation program is for DoD to continuously 
monitor and track the safety, reliability, and performance of only selected munitions to 
ensure these munitions stockpiles are at their highest state of readiness. As previously 
mentioned, it is impossible to do QE on all munitions stockpiles. Munitions are added to 
the QE list only after a fleet or ashore incident has occurred. According to Mr. Roger 
Swanson, in FY08 and onwards he will receive a budget of $3M to support QE on only 
surface ordnance. This $3M is used for destructive testing/inspection and to conduct 
analysis to determine the service life and to revise the maintenance schedule, if 
necessary, for the munitions. Additionally, he requires an additional $1.5M a year for 
test equipment used in testing munitions stockpiles for bit checks. Bit checks only 
indicate whether the munitions are good or bad, i.e., up or down. If it indicates down, 


additional testing and maintenance is required to determine the failure. Testing 
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equipment does not indicate how the munitions will perform once launched. The 
destructive testing/inspection and analysis done produce the ultimate results desired to 


find out how munitions will perform when launched. 


Destructive testing/inspections on munitions, e.g., a missile’s rocket motor, are 
done every three years depending on the rocket’s propellant composition. According to 
Mr. Swanson, a complete QE costs $0.5M per rocket. This does not include the buying 
of the replacement rocket required to return the missile to a state of mission readiness. 
Again, depending on the rocket’s propellant composition, a rocket will cost the service 


between $50,000 and $250,000. 


Although munitions mishaps hardly ever occur afloat or ashore (0.1 incidents per 
year), the cost they can incur cannot be overlooked. In the 1994 munitions incident, the 
cost to DoD was over $1M for the destruction of the munitions magazine, $590,000 for 
the lost of munitions, and $400,000 for the clean-up efforts. This does not include the 


investigation cost to DoD to determine the cause of the explosion. 


D. “TO-BE” COST (A WORLD WITH ATOS) 

1. Ordnance Management Process 

The implementation of ATOS, as previously mentioned, is not meant to replace 
any of the current traditional business processes of Ordnance Management. Rather, 
ATOS is meant to enhance the current Inventory Management process and add additional 
capabilities to Ordnance Management, e.g., the system will be capable of automatically 
monitoring and reporting user defined environmental conditions experienced by the 
munitions in near real-time. Additionally, by implementing ATOS the number of labor 
intensive man-hours expended in accomplishing the current Inventory Management 


process discussed above, can be greatly reduced. 


The following is a description of how the six munitions management tasks of the 
Inventory Management process used in the MUA by AFOTEC can be improved when 
using ATOS so that the reduced workload can allow for other productive efforts to be 


accomplished [AFOTEC 2004]. 
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1. Munitions Receipt. When munitions arrive at a depot, munitions 
or logistics personnel attach RF tags to munitions pallets and populate RF tags with asset 
information for each munitions pallet using the HHR. Next, tagged munitions pallets are 
receipted and the asset information is stored on the HHR as a transaction record. Tagged 
munitions pallets are moved to a storage location where the RCU retrieves the asset 
information and the latest environmental data from the munitions pallets. The RCU 
updates and stores the RF tag data until the PP commands it to transfer the data via 
WLAN, LAN, or the HHR serial port. The PP reconciles the transaction records from the 
HHR and RCU data, updates the inventory list, and graphically displays the data on the 


computer screen. 


2. Inventory Maintenance. The RCU maintains the munitions 
inventory through periodic interrogations of the RF tags. If the RCU encounters a new or 
missing RF tag, it sends a flag to the PP. When conducting inventory tasks using an 
HHR, the ATOS operator first downloads the inventory list from the PP to the HHR. The 
operator then takes the HHR to the magazine. The HHR queries the RF tags and 
reconciles the inventory list with the tagged munitions pallets inside the magazine. Any 
new or missing RF tags are flagged by the HHR and associated data are later transferred 
to the PP. The operator takes appropriate actions to resolve any flags that were sent to the 
PP from the HHR or RCU. Throughout this process, the RF tags continually collect and 


store environmental data. 


a: Munitions Movement. When tagged munitions pallets are moved 
from one location to another within the same depot, the operator uses the HHR to update 
the RF tag location information. On the next interrogation, the RCU automatically 


updates location data on the PP inventory list. 


4. Munitions Issue. The operator issues one or more individual 
munitions items from tagged munitions pallets to a local field unit and uses the HHR to 
update the munitions count on the RF tag. The RCU captures the munitions count change 


and updates the PP inventory list. 


2. Munitions Transfer. The operator ships a tagged munitions pallet 


to a location outside of the depot, using the HHR to document the transfer. The operator 
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uses the HHR to download the latest asset information, environmental data, and any 
alarm flags. The HHR is docked to the PP and data are transferred to the PP. The RF tag 
ID number is removed from the inventory list and the data are archived. The PP 
automatically updates the inventory records of the losing depot when the HHR is docked 
to the PP. Munitions pallets are not tracked while in transit between depots, but the RF 
tags will continue to collect environmental data, which is transferred to the PP at the final 


destination. 


6. Munitions Quality Evaluation/Surveillance. The operator removes 
one or more individual munitions items from tagged munitions pallets for 
inspection/maintenance. QE personnel use the HHR to update the pallet condition code 
and history to reflect the status of the individual munitions items and what QE actions 
were taken upon its return to the inventory. The RCU captures the updated information 
and transfers it to the PP. The PP then reconciles the RCU and HHR data and updates the 


inventory list. 


2. Ordnance Management Process Cost 

There are many benefits in implementing an ATOS infrastructure. One of the 
main objectives is to reduce the cost of the current Inventory Management process. The 
initial fielding of ATOS will be the additional cost incurred in the first year. From the 
second year onwards, DoD will benefit from the dividends that ATOS provides in the 
inventory of munitions. For the dividends in the environmental, munitions environmental 
data needs to be collected for at least a ten-year time frame. The following describe how 


ATOS changes the current management of munitions. 


Manpower costs are expected to be reduced because personnel no longer have to 
do the wall to wall inventories that can take many man-hours. ATOS provides magazine 
inventories near real-time. Additionally, many man-hours are saved in the receipt of 
munitions because documents, such as DD1348-1 forms, will be obtained from the 


accountable officer and uploaded directly into the handheld readers. 


Demilitarization is another area in which DoD can save millions of dollars a year. 


There is currently no implemented method that identifies the condition of all munitions in 
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inventory. Quality Evaluation done in selected munitions is the only method that exists 
that identifies the serviceability and reliability of munitions. If munitions of a specific 
batch are identified as “bad” munitions, the entire batch manufactured on the same date 
are removed from inventory and taken to be demilitarized. This is also true for munitions 
that are accidentally dropped. If the dropped munitions tested “bad”, the entire pallet of 


dropped munitions is removed from inventory without further testing. 


The ability to gather environmental data munitions are experiencing or have 
experienced in the magazines, in transit, or in the field, can reduce the millions of dollars 
DoD is currently spending in corrosion maintenance. The environmental data collected 
can be used to change the environmental conditions under which the munitions are being 
stored. Additionally, this can assist personnel in revising the maintenance schedule of 
munitions stored under different conditions, i.e., in the field vice being stored in a 


munitions magazine. 


The collection of munitions temperature, humidity, and shock/vibrations will be 
used as input data for models to determine the munitions service life and to revise 
maintenance schedule of all munitions, instead of just the current munitions on the QE 


list. This ATOS initiative will significantly enhance the QE process. 


According to Mr. Swanson, ATOS has the potential to reduce QE expenditure by 
as much as 83% as there will be no need to conduct destructive testing/inspection of 
munitions as the QE program currently calls for. This does not mean that destructive 
testing/inspection will be terminated. This type of testing will still be required but over a 
longer period of time, i.e., ten years vice three years, to validate and improve the working 


models. 


The cost of developing and validating models depends on the models’ complexity, 
and the cost is comprised almost exclusively of labor costs which come with a decent 
price tag. I anticipate that the type of model that will be required to assist in QE of 


munitions will cost about $1.5M annually. 
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E. METHODOLOGY FOR ASSESSING CHANGES DUE TO ATOS 
IMPLEMENTATION 


For each Element in the ATOS Work Breakdown Structure, a judgement is made 
on what percentage reduction in the As-Is costs is caused by the implementation of 
ATOS. These percentage reductions will be SME-based for this thesis and pilot-project- 


data-based in the future. 


The reason these percentage reductions are SME-based is because there has not 
been an ATOS pilot program implemented long enough to collect sufficient data to 
accurately annotate the realistic reductions. The ATOS pilot programs implemented thus 
far have been programs implemented at a lower scale to collect data to conduct, e.g., a 
Military Utility Assessment (MUA). These programs have been tailored to accomplish 


specific objectives or answer questions posed by DoD decision makers. 


The Subject Matter Experts consulted in the process of thesis, have been 
personnel who have been actively involved in one way or another with ATOS and are 
currently Program Mangers at NSWC, Indian Head, or Ordnance Managers/Supervisors 
at Seal Beach Weapons Station. Ordnance Managers being both extremely familiar with 
the ordnance inventory management processes at their munitions sites and being actively 
engaged in the ATOS initiative are the most qualified personnel to perceive what steps in 
the ordnance process can be automated with ATOS implementation and what percentage 
reduction is realistic in the process. Of course, these are estimates and vary from 
munitions site to munitions site depending on the amount of munitions stockpiles being 


accountable for. 


In the inventory management process, these percentage reduction estimates are 
most confident because most work done thus far in ATOS has been in this area of 
interest. The percentage reduction estimates in the environmental area of interest, are not 
with a high degree of confidence. This is because pilot programs have not been 
implemented to collect munitions environmental data to support these estimates. All 
SMEs concur that it can take 10 to 12 years of munitions environmental data collection 


before a rational data-supported estimate can be achieved. 
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The percentage reduction estimates provided by the SMEs as stated above, are our 
initial estimates that need to be validated by the results from pilot projects. This will 
occur as SMEs over time have collected sufficient data to analyze and recommend more 
precise estimates. At that point, the savings ATOS provides will be embraced by the 


services that are not now providing their full support. 


Table 2 below displays the percentage reduction estimates that are used in this 


analysis. 


F. COMPARISON OF “AS-IS” TO “TO-BE” 


Two Return on Investment analysis are presented. 


e The first ROI is on the impact of an investment in an ATOS infrastructure 
on the Inventory Management process and the successful management of 
DoD’s munitions stockpiles. We have access to some data for this 
analysis, and the benefits from this investment are available to the analysts 


relatively quickly. 


e The second ROI focuses on the impact of an investment in ATOS in the 
arena of munitions environmental data. This is more theoretical because 
data needs to be collected and analyzed over a relatively long period of 


time in order to address effectively the savings that ATOS can provide. 


1 Inventory Management Process Return on Investment Analysis 

To do the first ROI analysis of measuring the impact of an investment in ATOS 
infrastructure on the Inventory Management process and the successful management of 
DoD’s munitions stockpiles implementing ATOS , I had to gather cost estimates for the 
Operation & Support to capture the As-Is cost. For clarification, the As-Is costs are the 
costs that are occurring in the current situation, which is a world without the 
implementation of ATOS. To determine the To-Be cost (that is, the costs that would 


occur in a future situation, a world with implementation of ATOS I required 
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e Cost estimates for the initial investment to put ATOS in place, and 


e Percentage reductions in Operation & Support costs that are due to the 


implementation of ATOS. 


Since ATOS has not yet been implemented, I was able to gather just cost 
information that has been expended thus far or planned to be expended to implement a 
portion of ATOS. An example of this kind of expenditure is for, the inventory 
management process. To develop, or forecast, the To-Be cost proved to be a challenge, 
especially when it came to the environmental because it requires the collection of 
munitions environmental data over time to determine effectively the percentage 
reduction. Only the implementation of an ATOS pilot aimed to capture munitions 
environmental data over a long period of time will identify a sensible percentage 
reduction. After many discussions with SMEs currently working in the ATOS project, I 
was able to get cost estimates that I could use to proceed with the ROI analysis. Cost 
estimates for other portions of the analysis were developed on a best guess basis. Table 2 
below displays the cost estimates used in the first ROI analysis for a notional five 
munitions magazine. It should be noted that the implementation of these five magazines 
is assumed to be what would be needed for four Hellfire Missile magazines and one 


Standard Missile magazine. 
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* Hardware/Installation 
* Modelling Validation (1) 1.250 


| * Munitions InventoryManagement{ 1,200] 50% | 000 


* Mishap Cost 204.6 102.3 102.3 
* Quality Evaluation Cost 
* Corrosion Maintenance Cost 537.5 268.8 268.8 


(1) Modelling Validation cost of $1,250K is spread over a two-year period: $625K/Year. 





Table 2. Estimates Used in the ROI Analysis for a Notional Five Munitions 
Magazine (All Estimates are in $K). 


There is no formal Navy guidance that lays out the initial investment of the 
implementation of an ATOS infrastructure at a munitions facility because a fully 
implemented ATOS infrastructure has yet to be implemented. What does exist are costs 
of partially implemented ATOS scenarios that were done to answer questions posed by 
DoD elements, such as the Air Force Operational Test and Evaluation Center (AFOTEC) 
which conducted a Military Utility Assessment of ATOS to determine its utility to the 


warfighter. The report from this assessment may be found in Appendix B of this thesis. 


In a 30 July 2004 Memorandum, the Office of Secretary of Defense (OSD) Under 
Secretary of Logistics directed the implementation of RFID across the Department of 


Defense with full implementation to be competed in FY07. In an effort to meet the 
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dateline, OPNAV N4 has taken the lead in the research and identification of a robust 
RFID capability, which will provide the highest Return on Investment and that requires 
minimal installation and out year maintenance costs. OPNAV has identified two Naval 
Weapons Stations, Seal Beach, California, and Yorktown, Pennsylvania, to implement an 
ATOS pilot. Phase I of this pilot will demonstrate how inventory can be accomplished in 
real time with a very high degree of accuracy and track location of sub-assembly 
components of high value missiles. The extent of this implementation will be to outfit 
four Hellfire Missile magazines and one Standard Missile magazine with the capability to 
provide real time inventory and asset data visibility to Ammunition Distribution & 
Control (AD&C). Additionally, this part of the test will demonstrate the potential ability 


to change current supply chain management within and between activities. 
To capture the As-Is cost, I used a combination of the following: 
e FY00 Army Demilitarization Cost 
e Prior NPS work 
e 1994 Incident Cost Report 
e Estimate to outfit five munitions magazines 
e FY04 USAF Corrosion Maintenance Cost Report 
e SMEs best guess cost estimates 


SMEs want to make it clear that the figures provided are estimates and these 


estimates are different from munitions facility to munitions facility. 


To capture the To-Be cost of the Inventory Management process was a much 
harder problem. This is because, as previously mentioned, there is not enough data to 
determine the percentage reduction that ATOS would provide. The percentage 
reductions that I used are best guess estimates and need to be validated once data is 


collected and analyzed from a pilot project. 
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a. Baseline Operation & Support ROI Analysis 
This ROI analysis model is used as the baseline for the Operation & 
Support of a notional five munitions magazine at a Naval Weapons Station. In this 


baseline model, data is taken from Table 2 above, which includes 


e $1.327M to implement an ATOS infrastructure in this notional five munitions 


magazine, time phased as follows: 
o Year | - $0.712M 
o Year 2 - $0.625M 
e $1.275 annual costs for the Operation & Support 
To account for inflation, I used the inflation indices provided by the Naval 


Cost Analysis Division (NCAD). These indices are available at 
http://www.ncca.navy.mil/service/inflation.cfm. 


For net present value computations, I assumed a 10% discount on the 
savings produced every year starting. These savings begin in YR 1, under the assumption 


that implementation of ATOS occurs in YR 0. 


Figure 8 below displays the cumulative Operation & Support and 
Investment cost for the As-Is and the To-Be for the management of the five notional 
munitions magazines. Compared to the amount invested, I calculate a break even point in 
24 months. That is, the notional five munitions magazines at the Naval Weapons Station 


will obtain full return on the initial investment in 24 months. 
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Figure 8. Cumulative Operation & Support and Investment Cost. 


Figure 9 below displays the Net Present Value (NPV) of Operation & 
Support and Investment savings. I assumed a 10% discount on the savings produced 
every year starting on YR | since I am assuming implementation of ATOS in YR 0. 
With these assumptions, I calculate that the notional five munitions magazines at the 
Naval Weapons Station can save approximately $5.2M (undiscounted savings) and 
$2.8M (discounted savings) in a ten-year period by implementing ATOS in the Inventory 
Management process. The Return on Investment is calculated at 214 %. That is, if 
$1.3M was invested for 10 years at 10.8% compounded interest, one would save $2.8M 


which is 214% over the 10-year period, or almost 11% annual, compounded ROI. 
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Figure 9. Net Present Value of Operation & Support and Investment Savings. 


b. Baseline Environmental ROI Analysis 

This ROI analysis model is used as the baseline for the Environmental cost 
of a notional five munitions magazines at a Naval Weapons Station. In this baseline 
model, I assumed it would cost $1.3M to implement an ATOS infrastructure in the 
notional five munitions magazines. It currently costs the notional Weapons Station 
$0.8M due to environmental conditions. To account for inflation, I used the inflation 
indices provided by the Naval Cost Analysis Division (NCAD). I assumed a 10% 
discount on the initial investment every year starting on YR 1 since I am assuming 


implementation of ATOS in YR 0. 


Figure 10 below displays the cumulative environmental costs for the As-Is 
and the To-Be for the management of the munitions in the five notional munitions 
magazines. Compared to the amount invested, I calculate that the notional five munitions 
magazines at the Naval Weapons Station will obtain full return on the initial investment 


in 38 months. 
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Figure 10. Cumulative Environmental and Investment Cost. 


Figure 11 below displays the Net Present Value (NPV) of Environmental 
and Investment savings. I assumed a 10% discount on the savings produced every year 
starting on YR 3 since I am assuming implementation of ATOS in YR 0 and the 
collection of munitions environmental data for two years to collect the munitions 
environmental data required for analysis. With these assumptions, I calculate that the 
notional five munitions magazines at the Naval Weapons Station can save approximately 
$2.7M (undiscounted savings) and $1.3 (discounted savings) at the end of YR 10 The 
Return on Investment is calculated at 99 %. That is, if $1.3M was invested for 10 years 
at 10.8% compounded interest, one would save $2.7M which is 99% over the 10-year 


period, or almost 7% annual, compounded ROI. 
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Figure 11. Net Present Value of Environmental and Investment Savings. 


Cc. Baseline Cumulative Total ROI Analysis 

This ROI analysis model is used as the baseline for the combination of 
Operation & Support and Environmental costs of munitions at the Naval Weapons 
Station. In this baseline model, I assumed it would cost $1.3M to implement an ATOS 
infrastructure in a notional five munitions magazine Naval Weapons Station. It currently 
costs the Weapons Station $2M for the Operation & Support of the five magazines. To 
account for inflation, I used the inflation indices provided by the Naval Cost Analysis 
Division (NCAD). I assumed a 10% discount on the savings produced every year 


starting on YR | since I am assuming implementation of ATOS in YR 0. 


Figure 12 below displays the cumulative total cost for the As-Is and the 
To-Be for the management of the five munitions magazines. Compared to the amount 
invested, I calculate that the notional five munitions magazines at the Naval Weapons 


Station will obtain full return on the initial investment in less than one year. 
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Figure 12. Cumulative Total Costs. 


Figure 13 below displays the Net Present Value (NPV) of cumulative total 
savings. I assumed a 10% discount on the savings produced every year starting on YR 1 
for the Operation & Support and YR 3 for Environmental since I am assuming 
implementation of ATOS in YR 0. With these assumptions, I calculate that the notional 
five munitions magazines at the Naval Weapons Station can save approximately $9.2M 
(undiscounted savings) and $5.3M (discounted savings) at the end of YR 10. The Return 
on Investment is calculated at 403 %. That is, if $1.3M was invested for 10 years at 
10.8% compounded interest, one would save $5.3M which is 403% over the 10-year 
period, or almost 18% annual, compounded ROI. 
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Figure 13. Net Present Value of Cumulative Total Savings. 


2. Theoretical Environmental Return on Investment Analysis 

As with the case with Inventory Management, we do not have historical data on 
what percentage reductions in Operation & Support costs ATOS implementation would 
provide. We do not have any data to substantiate such an estimate, so we used 
percentage reductions that SMEs believe that ATOS can provide over time. For those 
estimates that SMEs were not comfortable in providing, we provide an estimate based on 


our professional judgment coupled with the discussions we had with SMEs. 


SMEs claim that it would take approximately ten years of environmental 
munitions data collection to begin analysis on the data collected. We are in the analytical 
position of the previous paragraph. One thing is for certain; all SMEs are in agreement 
that ATOS has potential savings in Quality Evaluation, Demilitarization, and Corrosion 
Maintenance. As previously mentioned, Mr. Roger Swanson, is confident that ATOS can 
reduce QE expenditures as much as 83%. Additionally, ATOS would allow all munitions 
to partake in the QE process vice just the current selected few. Furthermore, if during the 
munitions QE process, the munitions undergoing testing are found to be faulty, the entire 
batch of munitions manufactured on the same date is removed from the inventory and 
taken to be demilitarized. The current QE process assumes that if a sample of a batch 


53 


manufactured on a specific date is bad, no further testing is done. The entire batch is 
removed from inventory to prevent personnel and property damages. With the analysis 
of the collection of munitions environmental data, QE personnel would be able to notify 
munitions sites what munitions to remove from inventory. Mr. John Backes, NSWC, 
Indian Head, Code E13A, says that the way of managing munitions in the future should 
be “management by individual instead of entire lots.” The following is an excerpt from 
an e-mail that recaps the points discussed with an SME on his support of ATOS. 


1) I fully support ATOS or a similar system that will provide good 
environmental data. 


2) Management by individuals instead of entire lots should be the way of 
the future. 


3) We can only test a small sample (usually every 3 years) of the entire 
population. We typically test about 30 samples and in many cases there 
are many more lots. We also do not have good information on the 
exposure of this sample and how well it represents the entire population. 


4) In the future, I think we can develop a model that quantifies how much 
damage occurs from each environmental stressor. The model would 
predict when the unit was approaching a critical level of cumulative 
damage. The units would then be removed from the population. Early 
removals would be tested to verify the accuracy of the model. For items 
that are susceptible to thermal damage, | day at 150 degrees might cause 
the same amount of damage as 20 days at 120 degrees or 1 year at 70 
degrees. 


5) The biggest benefit of this capability would be an improvement in our 
ability to accurately monitor and predict the quality of weapons and 
remove defective weapons before they become a safety or reliability 
problem. It is much better to predict and remove these weapons instead of 
finding the problem when the fleet attempts to use the item. 


In demilitarization, I speculate that DoD will be able to save as much as 90% 
because munitions will only be demilitarized when it is actually known for certain that 
munitions have exceeded their serviceability and pose a danger to personnel and 
property. According to Mr. John Backes, when munitions are dropped, there is no testing 
on the spot of the incident to determine the status of the dropped munitions. The dropped 
munitions need to be shipped to a facility where testing can be done to determine the 


status of these munitions. If one of the munitions is found to be damaged due to the 
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dropped, the entire pallet that was dropped is demilitarized. When this occurs, munitions 
need to be shipped to replace the dropped munitions costing DoD additional monies in 
shipment and munitions replacement, not to mention a possible delay in mission 
accomplishment. Although incidents involved dropping of ordnance do not occur 
frequently, I speculate that on the worst case, 50% reduction in cost can be saved if 
personnel know what dropped munitions have received damages and need to be 
demilitarized without further testing. This figure can only be validated over time with the 


collection of munitions environmental data. 


In corrosion control, I speculate the availability of environmental munitions data 
can reduce the corrosion maintenance expense by at least 50%. As _ previously 
mentioned, in FY04 it cost the USAF $43M in munitions corrosion maintenance. The 
access to information about the environmental conditions munitions are or have been 
exposed to can significantly assist maintenance personnel in revising munitions 


maintenance schedules to ensure maximum munitions serviceability and reliability. 


G. SENSITIVITY ANALYSIS 

Sensitivity analysis is a procedure to determine the sensitivity of the outcomes of 
an alternative to changes. If a small change in a parameter results in relatively large 
changes in the outcomes, the outcomes are said to be sensitive to that parameter. This 
may mean that the parameter has to be determined very accurately or that the alternative 
has to be redesigned for low sensitivity. Sensitivity analysis helps to determine a range 
of plausible inputs to be considered when there is uncertainty about the true value of an 


input. 


In this thesis the cost estimates used in the ROI analysis were based on personal 
professional judgment. They can be validated only when an ATOS pilot is implemented, 
and the data collected from this pilot are analyzed. The initial investment, for example, 
will not be the same at every munitions facility because the initial investment depends on 


the size of the facility and the extent of the ATOS implementation. For this reason, I 
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conducted a sensitivity analysis varying the initial investment and the percentage 
reductions in various Operating and Support WBS Elements, keeping all other variables 


the same. 


Table 3 provides the results of sensitivity analyses in which the value of the initial 
investment is varied. Recall that the baseline value for the initial investment is $77K. In 
these sensitivity analyses, the initial investment is raised, in a step-wise way, to $200K, 
then to $400K, and finally to $600K. This latter figure represents an expansion of almost 
ten times the baseline investment costs. Table 3 displays the time in months when the 
notional Naval Weapons Station will reach the break even point, that is, the time it takes 
to obtain full return on the initial investment in Operation & Support given initial 
hardware cost of $200K, $400K, and $600K respectively. Note that while the cost of 
investment expands by almost ten times the baseline investment costs, the break even 
time expands by much less, not even doubling. This suggests that the attractiveness of an 


ATOS investment is very robust. 





Table 3. Payback Periods in Operation & Support. 


To satisfy the optimistic and pessimistic use of SMEs about the Operation & 
Support savings that ATOS can provide, I increased and decreased the percentage 
reductions in the Operation & Support keeping all other variables constant, to calculate 


the savings and the ROIs within the same time frames. 


Table 4 below displays the percentage reductions that I used in the sensitivity 


analyses | and 2. 
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« Munitions Inventory Management Process Cost 
* Demilitarization/Disposal Cost ee ee 





Table 4. Percentage Reductions in Operation & Support. 


Table 5 displays the To-Be cost, savings, the payback period, and the ROI for 


each case presented in Table 4. 











Table 5. Sensitivity Analysis on Changes of the Percentage Reduction in Operation 
& Support. 
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IV. OBSERVATIONS AND RECOMMENDATIONS 


OBSERVATIONS 

e Throughout this thesis, I have emphasized many times that ATOS is not a 
replacement for the current policy of ordnance management. Rather, the 
concept of ATOS is to enhance the current ordnance management processes 


for the effective management of DoD’s large munitions stockpiles. 


e Technology required to implement ATOS is COTS technology. There is no 
time delay in obtaining the hardware for full implementation. Of course, as 


requirements and security for ATOS changes, 


e The methodology for doing ROI analysis associated with the implementation 


of ATOS is well understood, and, with appropriate data, can be developed. 


e The data to support ROI analysis associated with the implementation of 
ATOS are not fully available, and they need to be developed. In some cases, 


these data need to be developed over multi-year time frames. 


e Based on projects done thus far (e.g., the Military Utility Assessment by 
AFOTEC (Appendix B)), ATOS can provide the capability to enhance the 
effective management of the large Department of Defense munitions 
stockpiles. See the Recommendations section below for the logical follow-on 


to this observation. 


e Mr. Roger Swanson and Mr. Mark Mentikov, Subject Matter Experts, do warn 
the decision makers having the final decision in making ATOS a program of 
record, that ATOS savings and benefits will not be reaped in all areas of 
interest simultaneously. Rather, the longer the system is implemented, the 
more attractive will the dividends be, especially in the environmental data 
collection. See the Recommendations section below for the logical follow-on 


to this observation. 


e Subject Matter Experts claim that there is not enough data on hand to 


determine the savings with ATOS if the environmental condition munitions 
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have experienced was known. See the Recommendations section below for 


the logical follow-on to this observation. 


Millions of dollars are spent every year in corrosion maintenance and Quality 
Evaluation to ensure munitions are mission ready. These millions of dollars 
can be reduced if QE personnel had better knowledge of the status of all 
munitions in inventory. This is not currently the case. For this reason, the 
RFID tags attached to munitions pallets need the capability to collect the 
environmental data that is required by QE personnel to input into models to 
provide Ordnance Managers valuable information about the status of their 
munitions in inventory. The more information is known about the condition 
of the munitions in theater, the less number of munitions, due to their 


reliability, will be needed to accomplish the warfighters’ assigned missions. 


B. RECOMMENDATIONS 


A complete implementation of ATOS to capture munitions environmental 
data to be analyzed in order to predict the serviceability and reliability of 
munitions. Currently no personnel know what percentage reduction ATOS 
can provide; one could only speculate. Only a pilot when implemented for a 
long period of time, say 10 years, can validate our speculations. It is critical 
that a commitment in implementing an ATOS pilot be made soon to 


commence collecting the benefits. 


A full scale ATOS pilot project at a munitions location, such as, Seal Beach, 


needs to be funded and implemented in order to 


collect the required data in order to validate the results from previous analysis, 


and 


support and provide a better argument as to why ATOS is the way to go in 


Ordnance Management. 


To establish an ATOS implementation plan DoD wide, a study on optimizing 


the placement of ATOS needs to be initiated. This will help decision makers 
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in determining the optimal locations that provide the greatest savings due to 


the large munitions stockpiles. For example, small munitions facilities where 


munitions do not have a high cycle rate may not be a high priority for the 


implementation of ATOS. 


e RFID tags attached to munitions pallets need the capability to collect the 


environmental data that is required by QE personnel to input into models to 


provide Ordnance Managers valuable information about the status of their 


munitions in inventory. 


c: RECOMMENDATIONS FOR FUTURE RESEARCH 


This thesis provides an initial glimpse of the Return on Investment analysis of the 


savings and benefits that ATOS has the potential of providing the Department of Defense 


in managing its large munitions stockpiles. While the purpose of this thesis was to 


provide a methodological approach for conducting a Business Case Analysis for the 


ATOS ACTD, there are many opportunities to capture other complexities and variability 


associated with ATOS, including the following. 


It would be beneficial to the Department of Defense to know an ATOS 
implementation location plan to guarantee savings and benefits at the 
earliest stages of the ATOS implementation. An optimization model 
would provide the order in which munitions sites are fielded to ensure the 


sites with the highest cycle rate of munitions are implemented first. 


Although models may already exist that provide limited predictability 
about the conditions of only certain munitions, robust models will be 
required to process the environmental munitions data collected to 


effectively calculate munitions service life and reliability. 


Support and participate in the implementation of the pilot project and 
collect the data required to conduct a Business Case Analysis to further 


assist the decision makers decision in making ATOS a program of record. 
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APPENDIX A. ACTDS/JCTDS FY95 — FY 05 


FY2005 ASAP Actionable Situational Awareness Pull 
FY2005 CUGR Chemical Unmanned Ground Reconnaissance 


FY2005 COSMOS Coalition Secure Management and Information System 


FY2005 Epidemic Outbreak Surveillance 
FY2005 Joint Coordinated Real Time Engagement 


FY2005 JEERCE Joint Enhanced Explosion Resistant Coating Exploitation 
FY2005 Joint Force Projection 
FY2005 MSAT Medial Situational Awareness in Theater 


FY2005 Rapid Airborne Reporting & Exploitation 
FY2005 
FY2005 Sea Eagle Sea Eagle 


FY2005 SLED SOCOM Long Endurance Demonstrator 


FY2004 CORSOM Coalition Reception Staging & Onward Movement 


FY2004 COSINE Coalition Shared Intel Network Environment - DEMO COMPLETE 


FY2004 FTTS Future Tactical Truck System 

FY2004 JPADS Joint Precision Airdrop System 
FY2004 J-USC2 Joint Unmanned Sys Common Control 
FY2004 MANPACK MANPACK 


FY2004 MAJIIC Multi-Sensor Aerospace/Ground Joint ISR Interoperability Coalition 
FY2004 MAGNUM MAGNUM 


FY2003 Adaptive Joint C4ISR Node - DEMO COMPLETE 
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Joint Blue Force Situational Awareness - DEMO 
FY2003  |JBFSA COMPLETE 
FY2003 Midnight Stand (prev. IT) - DEMO COMPLETE 
FY2003 {|NVCUA Night Vision Cave & Urban Assault - DEMO COMPLETE 


FY2003 
FY2003 Tactical IFSAR Mapping 


FY2002 Boundary Step - RESID COMPLETE 


Coalition Information Assurance Common Operational 
FY2002 _|CIA COP Picture 


Contamination Avoidance at Seaports of Debarkation - 
FY2002 |CASPOD DEMO COMPLETE 

Expendable Unmanned Aerial Vehicle - RESID 
FY2002 _|EUAV COMPLETE 

Homeland Security Command and Control - DEMO 
FY2002 _|HLSC2 COMPLETE 
FY2002 |HYCAS Hyperspectral Collection and Analysis 
FY2002 |JDSR Joint Distance Support & Response - DEMO COMPLETE 
FY2002 |JEOD-KTOD Joint Explosive Ordnance Disposal - DEMO COMPLETE 


Language and Speech Exploitation Resources - DEMO 
FY2002 |LASER COMPLETE 


FY2002 Micro Air Vehicle 
FY2002 [PathF = —_| Pathfinder 
FY2002  |SIGINT SIGINT Processing 
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FY2002 |Space-Based MTI  |Space-Based MTI 
FY2002 |SPARTAN SPARTAN 
FY2002 Thermobarics - DEMO COMPLETE 


FY2001 Active Network Intrusion Defense - DEMO COMPLETE 
FY2001 Adaptive Battlespace Awareness - RESID COMPLETE 


FY2001 Advanced Tactical Laser 
D 


Advanced Technology Ordnance Surveillance - R] 
FY2001 |ATOS COMPLETE 


FY2001 |ACMD Area Cruise Missile Defense - RESID COMPLETE 
FY2001 CCID Coalition Combat ID - DEMO COMPLETE 
FY2001 Coalition Theater Logistics - RESID COMPLETE 


FY2001 Coastal Area Protection System - RESID COMPLETE 


FY2001 |{HSKT Hunter Standoff Killer Team 
FY2001 Joint Area Clearance - RESID COMPLETE 
FY2001 Loitering Electronic Warfare Killer 


| rv focr eesti, "wevocownete 
FY2001 = |NCCT NCCIS&R) - DEMO COMPLETE 
Personnel Recovery Extraction Survivability aided by Smart 


FY2001 _|TACMS-P Tactical Missile System Penetrator - RESID COMPLETE 


FY2001 {TIPS Theater EET nna Planning System - DEMO COMPLETE 
FY2000 |CINC 21 ICINC21-RESIDCOMPLETE sd 21 - RESID COMPLETE 


Oalition Aerial Surveillance and Reconnaissance - Rl 
FY2000 _|CAESAR COMPLETE 


FY2000 |C/NOFS Comm/Nav Outage Forecast System 


Computerized Operational MASINT Weather - RESID 
FY2000 |COMWX COMPLETE 


FY2000 |Umbrella) Content-Based Info Security 


Global Monitoring of Space ISR Systems - RESID 
FY2000 _|GMSIS COMPLETE 
FY2000 Ground-to-Air Passive Surveillance - RESID COMPLETE 


Joint Intelligence, Surveillance & Reconnaissance - RESID 
FY2000 |JISR COMPLETE 
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FY2000 {MLAS Multiple Link Antenna System - RESID COMPLETE 
FY2000 Quick Bolt - RESID COMPLETE 
FY 2000 Restoration of Operations - RESID COMPLETE 


Battle Damage Assessment in the Joint Targeting Toolbox - 
/oherent Analytical Computing Environment - RESID 
ONPLETE 
ommon Spectral MASINT Exploitation - RESID 


a 
FY1999 |CEASE II COMPLETE 
Force Medical Protection / Dosimeter - RESID 


Human Intelligence & Counterintelligence Support Tools - 
FY1999 —|HICIST RESID COMPLETE 
Oit Medical Operations 7 Telemedicine - 
FY1999 — |JMOT COMPLETE 


FY1999 = | JTL (JLCP) Joint Theater Logistics - RESID COMPLETE 


Personnel Recovery Mission Software - RESID 
FY1999 |PRMS COMPLETE 


FY1999 Small Unit Logistics - RESID COMPLETE 
| D 


eater Air & Missile Defense Interoperability - 
FY1999 —|TAMDI COMPLETE 


FY1998 {ACOA Adaptive Course of Action - RESID COMPLETE 
FY1998 — |C4I for CW C4I for Coalition Warfare - RESID COMPLETE 
FY 1998 High Power Microwave - RESID COMPLETE 


Info Assurance: Automated Intrusion Detection 
FY1998 IA:AIDE Environment - RESID COMPLETE 
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Joint Biological Remote Early Warning System - RESID 
FY1998 — |JBREWS COMPLETE 


Migration Defense Intelligence Threat Data System - 
FY1998 | |MDITDS RESID COMPLETE 
FY 1998 Precision Targeting Identification - RESID COMPLETE 


Space Based Space Surveillance Operations - RESID 
FY1998 —_|SBSSO COMPLETE 
FY1998 _|TPSO Theater Precision Strike Ops - RESID COMPLETE 
FY1998 Unattended Ground Sensors - RESID COMPLETE 


FY1997 {Chem Add-On Chemical Add-On to Bio Detection - RESID COMPLETE 


FY1997 Consequence Management - RESID COMPLETE 
FY1997 {CP II Counterproliferation II - RESID COMPLETE 
FY1997  |ELB/JTFW COMPLETE 


cs IOPT oe Info. ae ee Planning Tool - RESID COMPLETE 


| Fy1907 | ICM integrated Collection Mgt.- RESID COMPLETE _—| Collection Mgt. - RESID COMPLETE 


Joint Advanced Health and Usage Monitoring System - 
FY1997 | JAHUMS DEMO COMPLETE 


FY1997. |MOUT(MOBA) — |Military Ops in Urban Terrain - RESID COMPLETE 
FY1997 _|RTV (RBV) Rapid Terrain Visualization - RESID COMPLETE 
FY1996_ _|ABP Bio Det Airbase/Port Biological Detection - RESID COMPLETE 
FY 1996 COMPLETE 
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MAI 
FY1995 —_|(Predator) Med Alt Endurance UAV - RESID COMPLETE 
FY1995 —_|PSTS Precision SIGINT Targeting Sys. - RESID COMPLETE 
FY1995_|P/RC-MRL Precision/Rapid Counter-MRL - RESID COMPLETE 


RITI (11-SUK, 
FY1995 |EFOGM) Rapid Force Projection Initiative - RESID COMPLETE 
FY1995_—_|(JASP) Synthetic Theater of War - RESID COMPLETE 
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EXecUtive SUMMALY® scsissccecccessscicccescceaccecksescuse 1 


The Advanced Technology Ordnance Surveillance Advanced Concept 
Technology Demonstration was intended to demonstrate an automated 
system that could be used to enhance the management of munitions and 
provide near real-time environmental surveillance data on stored critical 
muuutions. This document describes the results. conclusions. and recom- 





LUE C201) eee ae ee oe Pe 3 


==> Detachment 1 Air Force Operational Test and Evaluation Center conducted 
a military utility assessment to determine the capability of the Advanced 
Technology Ordnance Surveillance system to provide reliable and accurate 
=~ environmental surveillance data on tagged mumuitions pallets and to support 
4 === muumitions management tasks. Data were collected from a technical 
_ see evaluation at the Naval Surface Warfare Center, Indian Head, MD. and 
a from three operational demonstrations at Miesau Anny Ammunition Depot. 
Miesau, Germany; Crane Anny Ammunition Activity Depot, Crane, IN; and 
Norfolk Naval Station, Norfolk, VA, on two US Navy vessels. These 
events were conducted between March 2003 and August 2004. 






I RIP OI cscs cicacisctsecnsipiecinstpiasataeaiipe ee mencenieanbence ll 


The Technical Manager provided 19 detailed scenarios that were used to 
assess how well the Advanced Technology Ordnance Surveillance system 
supported the munitions management tasks at strategic and tactical depots. 
Objective data were collected to evaluate system performance in a 
Tealistic operational environment and to determine if the munitions system 
met US European Command functional requirements. Questionnaires 
were provided to the warfighters to capture their opimions of the system's 
TROUT sac ares ccs pecerem ation cacmececctengncnpmnneasete 17 


The Advanced Technology Ordnance Surveillance system demonstrated 
the capability to enhance the mumitions management logistics processes 
and provide the warfighter with environmental surveillance information on 
tagged munitions pallets. Questionnaire responses and comments from 
warfighters indicated the system was effective in providing environmental 
surveillance information on tagged nmumitions pallets and enhanced the 
nuutitions management tasks. 








Conclusions and Recommendations .........23 


The Advanced Technology Ordnance Surveillance system demonstrated 
military utility, but could be improved by enhancing handheld reader data 
transfer capability to support up to 1,000 records. modifying the radio 
frequency tag so munitions quality assurance data can be entered, and 
evaluating the interoperability of the system with service ammunition 
automated information systems. 
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EXECUTIVE SUMMARY 


The purpose of the Advanced Technology Ordnance Surveillance (ATOS) Advanced Concept Technol- 
ogy Demonstration (ACTD) was to demonstrate an automated system that was designed to provide 
neat-real time environmental data on critical mumutions and use the latest n automatic identification 
technologies (AIT) and micro-electrical mechanical sensors (MEMS) to enhance munitions manage- 
ment. This document presents the results, conclusions, and recommendations denved by Detachment 1 
(Det 1) Aur Force Operational Test and Evaluation Center (AFOTEC) for the military utility assessment 
(MUA) of ATOS. 

The ATOS MUA includes results obtained during a technical evaluation (TECEVAL) and three opera- 
tional demonstrations conducted from March 2003 to August 2004 at four locations. The TECEVAL 
was held at Naval Sea Systems Command (NAVSEA), Indian Head, MD, between 28 March 2003 
and 16 July 2004 to assess ATOS performance in a controlled laboratory environment. The Det 1 
AFOTEC assessment team collected performance data to augment data collected dunng the opera- 
tional demonstrations. The first operational demonstration was held at Miesau Anny Munitions Depot, 
Miesau, Germany from 9 to 23 March 2004 and assessed ATOS’s capability to support distibution of 
muuutions from a theater-level depot. The second operational demonstration was held at Crane Amy 
Ammunition Activity (CAAA), Crane, IN, from 12 to 23 Apml 2004 and assessed ATOS’s capability to 
support wholesale distbution of munitions from a strategic-level depot. The third operational demon- 
stration was held at Norfolk Naval Station, Norfolk, VA, and consisted of two events. The first event 
was held on the United States Ship (USS) Ponce from 12 to 23 Apnl 2004. The second event was 
conducted in two parts: the first part was held on the pier where the USS Harry S. Truman was docked 
from 17 to 19 August 2004, and the second part was held on the USS Harry S. Truman from 23 to 27 
August 2004. These three maritime events assessed ATOS’s capability to support munitions distribution 
aboard naval vessels. 


As directed by US European Command (USEUCOM), the MUA assessed ATOS’ capability to support 
environmental surveillance and munitions management. The environmental surveillance focused on the 
tadio frequency (RF) tag’s capability to collect and store temperature and relative humidity (RH) data. 
The munitions management focused on ATOS’ capability to support and enhance nuuuitions manage- 
ment tasks (pallet receipt, inventory maintenance, pallet movement, pallet issue, pallet transfer, and 
muuitions quality assurance [QA]). In addition to the USECOM fimctional requirements, Det 1 
AFOTEC characterized ATOS system capabilities in tems of training, component mggedness, power 
Tequirements, and installation requirements. A security assessment was planned but not performed due 
to lack of security expertise from the operators. 
The ATOS system demonstrated potential military utility by providing the warfighter near-real time 
environmental surveillance data and supporting five out of the six mmmition management tasks during the 
demonstrations. The RF tags measured and recorded temperature and RH throughout the demonstra- 
tions. Although the RF tags performed well dunng the demonstrations, some of the warfighters ex- 
pressed concem about the cost of the RF tag and the fact that they were not designed to be reusable on 
other muuutions pallets. Also, the gravity-shock (G-shock) sensor on the RF tag must be further devel- 
oped and tested. The warfighters indicated that ATOS enhanced munitions management tasks and that 
it was easy to install, user fiendly, easy to learn. and easy to operate. Data transfer problems between 
the handheld reader (HHR) and the preprocessor (PP) diminished effectiveness of pallet receipt and 
inventory maintenance tasks. The RF tag record requires additional fields to accommodate tracking of 
QA mspections. The ATOS system demonstrated good support for pallet movement, issue, and transfer 
tasks. 
System recommendations fall into two categories: environmental surveillance and munitions manage- 
ment. For the former, RF tags require three upgrades: time/date stamp of temperature and RH 
Teadings that exceed lmuts, reusable tags, and further development of G-shock sensors. For munitions 
management, recommendations focus on data transfer capabilities and the addition of RF tag fields to 
allow tracking of QA mspections. 
ee] 
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SITUATION 





Background 


The effective management of the large Department of Defense (DoD) mumitions stockpile 
has long been hindered by labor-intensive inventory methods and stockpile inaccuracies. The 
current process involves manual data entry using an inventory sheet to tack and monitor the 
flow of munitions. The loss of asset visibility due to stockpile inaccuracies also creates a host 
of problems for the acquisition manager, logistician, storage custodian, and the warfighter. 
Overall mission readiness is degraded because needed munitions either cannot be located, or 
when located. the serviceability cannot be determined because the current surveillance 
methods do not provide the cntical environmental information needed to determine munitions 
serviceability. Storage conditions. particularly temperature and RH profiles, and G-shock can 
have significant impacts on munitions’ safety and reliability. By utilizing the latest im AIT and 
MEMS. the warfighter should have a capability that provides near-real-time environmental 
data and automated support for managing the complex DoD munitions stockpile. 

The ATOS ACTD is a Department of the Navy ACTD sponsored by USEUCOM. 
USEUCOM requested Det 1 AFOTEC to conduct a MUA of the ATOS system. This report 
describes the assessment of ATOS performance using data from the TECEVAL and opera- 
tional demonstrations. It includes the assessment methodology, data collection methods, data 
analysis, and MUA results. 


System Description 


The ATOS system is an automated system that provides the warfighter a capability to collect 
environmental data in near-real-time and supports munitions management tasks on stored 
munitions pallets. The ATOS system consists of six major components: RF tag. HHR, Reader 
Control Unit (RCV), PP. environmental database (EDB). and an interface to service ammuni- 
tlon automated information system (AIS). In addition, radio frequency extenders (RFE) are 
used to enhance RF coverage inside enclosed magazines or depots. The EDB and interface 
to Service Ammumition AIS were not available to demonstrate during the ACTD. Figure 1 
shows comnectivity for the ATOS system components. 






Figure 1. ATOS System: ATOS was 
designed to provide environmental 
surveillance data and to support 
munitions management tasks. 
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The RF tag is a small enclosure attached to a nuumutions pallet using an RF tag holder. The RF 
tag contains temperature, RH, and G-shock sensors. The temperature and RH sensors collect 
environmental data according to a defined schedule (every 60 minutes during the ACTD). 

The RF tag also has a receiver, transmitter, and non-volatile memory to receive, transmit, and 
store wuque asset information and environmental data. 


The HHR is a portable wireless barcode scanning device with the capability to read and wnte 
asset data to and from the RF tags and read linear and 2-dimensional barcodes. Using a 
keypad and touch screen, the users can upload and download asset information to and from 
the RF tags using a function called roll call, which queries the RF tag to identify itself and 
provide any changes to asset and environmental data. Data are stored in the HHR as trans- 
action records until they are transferred to the PP for analysis and storage. The HHR can 
also retrieve stored RF tag data from the RCU using an Ethemet cable. The HHR uses a 
Windows Pocket Personal Computer operating system, which is a commercial off-the-shelf 
technology (COTS) software. 

The RCU is a fixed RF reader that collects asset and environmental data from the RF tags. 
The RCU is powered by 24 volts direct cwrent (VDC) and can be mounted inside a munitions 
magazine. Depending on the size of the magazine, RFEs are connected in series with the 
RCU to achieve optimal RF transmission coverage. The RCU obtains data from the RF tags 
using a function called interrogation, which consists of two independent reads (short and long). 
The short read determines the number of RF tags detected, RF tag identification (ID), Depart- 
ment of Defense Identification Code (DODIC), National Stock Number (NSN), consignee, 
any environmental sensor flags, and a low battery flag. The time duration for a short read 
depends upon the number of RFEs connected in series with the RCU, the number of RF tags 
detected inside the storage facility, and the type of material (wooden crates and aluminum 
pallets) on which the RF tags were attached. The long read downloads the remaining asset 
information and environmental data. The duration of a long read can take hours to days 
depending on the number of RF tags being interrogated. The RCU serves as a long-term data 
storage unit for receiving and storing interrogated RF tag data. The RCU stores these data 
until the PP commands it to transfer the data via a wireless local area network (WLAN), local 
area network (LAN), serial cable connection with the HHR. The RCU has no direct user 
interface (1.e., no keyboard or display). 

The PP is an interactive command and control system designed to retrieve RF tag data from 
the RCU and HHR. The PP is typically located in the central work area. Munitions informa- 
tion is formatted and passed from the PP to service ammunition AISs (e.g., Retail Ordnance 
Logistics Management System for US Navy local and theater inventory management of 
niunitions) and the EDB. Munitions experts and analysts can use the EDB to evaluate 
niunitions performance fluctuations due to environmental changes. 

System Concept 

The ATOS system provides two key capabilities to manage the DoD mumitions inventory 
process: environmental surveillance and munitions management. To perform the environmen- 
tal surveillance capability, the RF tag collects and records temperature and RH data from 
tagged munitions pallets. For the ACTD, the RF tags were programmed to collect and record 
temperature and RH data every 60 minutes. The G-shock sensor was not demonstrated for 
this assessment. The environmental data collected by the RF tags were downloaded to the 
PP via RCU or HHR. Table 1 describes the munitions management tasks, Figure 2 graphi- 
cally displays how ATOS automates these tasks. and Table 2 describes the ATOS automation 
of each task. 
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Table 1. Munitions Management Tasks: These six tasks form the foundation for inventory management 
of munitions. 








Figure 2. Automated Munitions Management Functions: ATOS is designed to automate the 
current labor-intensive munitions inventory management processes by supporting six 
munitions management tasks. 
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Table 2. Munitions Management Tasks using ATOS System: ATOS system automates the 
currently labor-intensive munitions management processes. 


Munitions Management Processes using ATOS 


When munitions amive at a depot, munifions or logisfics personnel attach RF tags to 


where the RCU retrieves the asset information and the latest: 

muniions pallets. The RCU updates and stores the RF tag data unl the PP commands itto 
transfer the data via WLAN, LAN, or the HHR serial port The PP reconciles the transaction 
prvapsal rpc ep a aa a 2 A 


Tagged munifons pallets are moved from one location  andther withn the same depot. The 
eperaor ues the HH to update the RF tag location infomation Onto nest intemogaton, 


peer ay a The operator uses the HR to downoad the latest anset 
any alam flags. The HHR is dockad to the PP and data 
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Assessment Overview 

Objective and subjective data were collected from the TECEVAL and three operational 
demonstration events between March 2003 and August 2004 to assess the ATOS system 
mulitary utility m providing environmental data and automating the munitions management 
tasks. Table 3 describes the TECEVAL and operational demonstrations used to assess the 
ATOS system. 


Table 3. MUA Demonstration Schedule: System performance was assessed during the TECEVAL 
and three operational demonstrations. 


Demonstration Schedule 





Contnenat Outed Save NEI) 

Limitations and Constraints 

While a thorough assessment of the ATOS system was desirable, it was constrained by two 

known limutations listed in Table 4 and five additional limitations listed in Table 5 encountered 
Table 4. Original Limitations and Constraints: These two limitations were well known prior to 
assessment team deployment. 


Original Limitations and Constraints 
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Table 5. Additional Limitations and Constraints: Five additional limitations and constraints 
were identified during the demonstrations. 


Pe stinfejet@@itotie-tuleje f-telet@te stee-tiels-] 


== revpemesaaiiaes ‘emperdureand | Used TECEVAL data to verily sensor accuracy. 
stamp information. measurements to a speciic ime. woe 











Participating Agencies 


US European Command 


The USEUCOM is a unified combatant command whose mussion is to main- 
tain ready forces to conduct the full spectrum of military operations, enhance 
transatlantic security through support to the North Atlantic Treaty Organiza- 
tion, promote regional stability, and advance US interests in Europe, Africa, 
and the Middle East. As the ACTD sponsor, the USEUCOM provided the 
Operational Manager (OM). developed operational requirements for the 
loyment of the ATOS program technologies. developed the concept of 
operations (CONOPS), and supported plans for the demonstrations. 


Joint Munitions Command 


The Jomt Munitions Command (JMC) serves as the Amy's single manager 
for producing, storing, maintaining, and demulitanzing conventional ammumition 
for all military Services. The JMC developed the Demonstration Master Plan, 
which articulated the ACTD demonstration strategy used to evaluate the 
ATOS system. Additionally. the JMC provided the Demonstration Manager. 
assisted USEUCOM in developing the ATOS system CONOPS and support 
plans, and provided operators for the CONUS demonstation. 
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Naval Sea Systems Command 


The NAVSEA, Naval Surface Warfare Center (NSWC). Indian Head Divi- 
sion, serves as the leading research and development orgamization specializing 
in developing products for special weapons, explosives, pyrotechnics, and 
ordnance handling. Its primary mission is to transform mulitary requirements 
into naval capabilities. The NAVSEA was responsible for the program 
management of the ATOS ACTD. During the MUA, the NAVSEA provided 
training resources, the support associated with the development and demon- 
stration of ATOS system technologies, technical support for ACTD demon- 
strations, and the ACTD TM. NAVSEA also provided the Transition Man- 
ager, who is responsible for transitioning the technology to the warfighter. 





US Air Forces in Europe 
The mission of US Air Forces in Europe is to plan. conduct, and coordinate 
offensive and defensive air operations in the European area of responsibility 
(AOR). The US Air Forces in Europe provided operators for the OCONUS  _ 
demonstration. 


T 


US Army, Europe 

The mission of US Army, Europe is to maintain forward-deployed combat 
forces trained and ready to conduct and support joint and multinational opera- 
tions and conduct engagement activities to protect US interests in the Euro- 
pean AOR. The US Army. Europe provided operators for the OCONUS 
demonstration. 





US Atlantic Fleet 


The US Atlantic Fleet provides fully trained, combat ready forces to support Sins 
US and North Adantic Treaty Orgamzation commanders in regions of conflict ge 

throughout the world. From the Adriatic Sea to the Arabian Gulf. Atlantic Fleet  * w F 
units respond to National Command Authority tasking. The US Atlantic Fleet Ap RS 
provided operators for the maritime demonstrations. Bo) oe 


Det | AFOTEC 
The Det 1 AFOTEC mission is to provide independent and tailored demonstra- 
tion and utility assessments of new technologies in support of Combatant 
Commander and other government decision makers for the purpose of rapidly 
fielding warfighting capabilities. During this ACTD, Det 1 AFOTEC was the 
i it assessment agency for evaluating the military utility of the ATOS 
system. Det 1 AFOTEC refined scenarios, developed the MUA assessment 
methodology. and wrote this final MUA report. 
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EXECUTION 





Methodology 


The initial assessment methodology was based on characterizing the mulitary utility of indi- 
vidual ATOS components. Det 1 AFOTEC developed cntical operational issues (COD), 
measures of effectiveness (MOE), and measures of performance (MOP) based on the 
Tequirements stipulated in the USEUCOM Functional Requirements Document (FRD). 
Table 6 lists the COIs and MOEs from the approved assessment execution document. 
During the Miesau demonstration, USEUCOM directed Det 1 AFOTEC to assess ATOS 
system capabilities in support of environmental surveillance and munitions management. The 
details on munitions management tasks were not available in time to revise the COIs, MOEs. 
and MOPs. Therefore, the assessment team correlated existing component-level measures to 
environmental surveillance and muuutions management capabilities. As a result, MOEs 1.2 
and 1.6 address environmental surveillance, MOEs 2.5 through 2.8 characterize the ATOS 
system, and the remainder of the MOEs address munitions management. Appendix B 
provides the correlation of the MOEs to environmental surveillance and individual munitions 
management tasks. A complete list of MOP results is contained in Appendix D, while 
Appendix E correlates MOP results with USEUCOM FRD requirements. 


Table 6. COIs and MOEs: Two COIs and associated MOEs provided a framework to assess 
the military utility of the ATOS system 


(OO) ree (@) a 


4. Does ATOS support] 1.1 Capability of ATOS system to provide container location data. 

assat visibility, 1.2 Capability of ATOS system t acourately report and maintain environmental 
data on containers. 

1.3. Capability of ATOS RF tags to communicate with RCU or HHR. 

1.4 Capability of ATOS HHR and RCU to communicate and transfer data to PP. 

1.5 Capability of ATOS PP to interoperate with Service Ammunition Als. 

1.6 Capability of ATOS PP to interoperate with EDB. 

1.7 Characterization of network bandwidth connectivity to ATOS PP. 

1.8 Capability of ATOS system protect data. 

1.9 Capability of ATOS system t provide sufficient dala storage. 

1.10 Characterization of ATOS RCU antenna coverage. 

1.11 Capability of ATOS RCU to detect RF tags at initial startup. 

1.12 Usability of ATOS HHR input devices. 

1.13 Capability of ATOS HHR to create RF tags and barcodes. 


Baliaty lachma ni 








2.2 Usability of ATOS HHR and RCU. 

2.3. Usability of ATOS PP. 

2.4 Ease of configuraton for ATOS components. 
2.5 Adequacy of ATOS training. 

2.6 Adequacy of ATOS componat ruggedness. 
7 Supportability of ATOS powerrequirements. 
2.8 Supportability of ATOS installation. 


Data Collection and Analysis 


The assessment team collected both objective and subjective data during the operational demon- 
strations using the scenarios described in Appendix C. Det 1 AFOTEC used Microsoft Excel to 
store and manage data collected during the operational demonstration events. 
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Objective Data 
Objective data came from the following sources: 
+ Assessment team observations (initial RF tag detection, RF tag communications with 
HHR/RCU/PP) 
* Kestrel 4000 portable weather station data (baseline temperature and RH data dunng 
operational demonstrations) 
+ ATOS performance data (RCU performance logs, environmental histogram data) 


* Laboratory instrumentation at NSWC during the TECEVAL (baseline temperature 
and RH data, battery test. RCU antenna coverage, noise vibration data) 


¢ Extracts from RCU data (interrogation time cycle, number of RF tags detected, PP 
communications with HHR/RCU) 


Subjective Data 


Subjective data consisted of warfighter and SME feedback (documented on questiomnaires) 
and assessment team comments (documented on observation logs). The Det 1 AFOTEC Test 
Director observed ATOS traming and the lead analyst observed the ATOS installation. Both 
the Test Director and lead analyst observed warfighters during operational demonstration 
scenarios. Questionnaires were tailored and administered to SMEs based on their area of 
expertise (operator, munitions, logistics. or secunty). Appendix F provides a complete listing 
of questionnaire results from all operational demonstrations. 


ATOS Events 


Technical Evaluation 


The purpose of the TECEVAL was to collect performance data in a controlled laboratory 
environment. The TECEVAL consisted of four visits to the NSWC, Indian Head. MD be- 
tween 28 March 2003 and 16 July 2004. During these visits, the assessment team collected 
performance data on the RF tag environmental sensors, antenna coverage range data, and 
changes to the PP software. Two end-to-end system tests were conducted to ensure ATOS 
would be ready to support the OCONUS demonstrations. A test of ATOS autonomous 
operation was conducted from 12-16 July 2004. 


OCONUS Demonstration 


The purpose of the OCONUS demonstration was to collect performance data at the largest 
theater-level munitions storage area in Europe. The OCONUS operational depot demonstra- 
tion was conducted at Miesau Amy Ammunition Depot at Miesau, Germany from 9 to 23 
March 2004. During the first week, the assessment team observed user training, as well as 
installation of the ATOS components in a 60 x 40 x 15 foot earth-covered magazine and a 70x 
200 foot open storage area called the ammunition supply point (ASP) (see Figure 3). After 
the installation, the TM placed 200 RF tags at both locations. The assessment team collected 
observation data and administered training questionnaires during the first week of the demon- 
stration. During the second week, the warfighters performed the scenarios listed in Appendix 
C. The assessment team collected observation data; administered operator, logistics, and 
munitions questionnaires; and collected RCU performance data. The assessment team was 
not able to collect theater-level QA data because a QA participant was not available to 
support the demonstration. 
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Figure 3. Miesau Earth-Covered Magazine and ASP: The OCONUS 
demonstration provided two distinct environments for the ATOS system. 






CONUS Demonstration 


The purpose of the CONUS demonstration was to collect performance data at a strategic 
wholesale depot that receives and ships conventional munitions to the Amy and Navy world- 
wide. The CONUS demonstration was conducted at two test sites at the CAAA depot at 
Crane, IN dunng 12 to 23 April 2004. The assessment team observed user traming, as well as 
installation of the ATOS components (six RFEs, RCU, PP. and WLAN) ina 100x 50x 10 
foot earth-covered magazine (see Figure 4). This demonstration site used a WLAN repeater 
on top of a water tower to establish a line-of-sight comnection to the PP. which was approxi- 
mately 1.5 miles away. The second ATOS test site was a 200 x 50 foot storage warehouse 
(see Figure 5). Due to the size of the 
storage warehouse and the large amount of 
conventional munitions stored inside this 
facility, the TM installed 12 RFEs to ensure 
maximum RF antenna coverage. The TM 
installed a RCU, PP. and WLAN inside the 
storage warehouse. The RCU was 
hardwired to the PP via Ethemet connection. 
Only 200 RF tags were used in the demon- 
stration based on the approval letters re- 
ceived from the Amny and Navy to meet 
Hazards of Electromagnetic Radiation to 
Ordnance (HERO) requirements. 

The assessment team collected observation 
data and administered training questionnaires 
during the first week of the demonstration. 
Dunng the second week of the demonstra- 
tion, the warfighters performed the 19 





Figure 4. CAAA Earth-Covered Magazine: ATOS was 
pthc * : installed in this earth-covered magazine in Crane, IN from 
scenarios listed in Appendix C. The assess- — }? r@ 23 April 2004. 

ment team collected observation data; 

administered operator, logistics, munitions, 

and traiming questionnaires: and collected RCU performance data. 
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Figure 5. CAAA Storage 
5 Warehouse: Twelve RFEs 
were required to provide 
complete antenna coverage 
in a large storage warehouse. 





Maritime Demonstration 


The purpose of the maritime demonstration was to collect performance data inside ship 
magazines. The maritime demonstration consisted of two events conducted at Norfolk Naval 
Station. The first assessment event was conducted on the USS Ponce from 6 to 9 July 2004 
to characterize system performance on a combat logistics ship. Since the magazine was very 
small (10 x 22 x 8 feet), an RCU was not installed (Figure 6). In this demonstration, the 
HHR was used to perform the six munitions management tasks. During this demonstration, it 
was leamed that an RCU was not required to perform munitions management tasks if a 
HAR and PP were available. The TM staff operated the PP since a warfighter was not 
available for training during the demonstration. The assessment team observed training and 
wartighter performance of munitions management tasks. At the end of the first maritime 
event, the assessment team admunistered questionnaires. 





4 


. ta - 
Figure 6. Close-in Weapons Support Storage on the USS Ponce: A HHR was used to detect 60 RF 
tags placed inside the storage facility. No RCU waz installed inside the magazine. 


; 
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The second maritime assessment event was conducted 17 to 27 August aboard the USS 
Harry S. Truman. This assessment was performed in two parts: the first part characterized 
the miggedness, reliability, and fimctionality of 32 RF tags that were attached to munitions 
pallets and shipped via commercial ground transportation from CAAA to Norfolk, VA; the 
second part charactenzed system performance on an aircraft carner (1.e., a combat ship). 
The second event installed the ATOS system im a 45 x 80 x 10 foot magazine below the flight 
deck hangar (Figure 7). The assessment team observed training and warfighter perfor- 
mance of munitions management tasks. At the end of the second maritime event, the assess- 
ment team administered questionnaires. 





Figure 7. Ship Magazine on the USS Harry S. Truman: A 
member of the TM staff installed wiring for the RFE: 
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RESULTS 


Results are presented for environmental surveillance, munitions management tasks, and 
system charactenzation. This will facilitate discussion of mulitary utility conclusions and 
recommendations. As stated earlier, MOEs 1.2 and 1.6 support environmental surveillance. 
MOEs 2.5 through 2.8 characterize the ATOS system, and the remainder of the MOEs 
support munitions management. The detailed results by MOE are discussed in Appendix D. 
Appendix E provides the correlation of FRD requirements to the MOP results. Appendix F 
depicts the correlation of the assessment questionnaires to environmental surveillance, 
munitons management. and MOPs. 





Environmental Surveillance 


Data gathered dunng the TECEVAL indicated that the temperature 
sensor was accurate to within + 1° Fahrenheit (F) of the environ- 
mental chamber settings. and that the RH was accurate to within = “Nice to see the temperature 
10%. These data further indicated that the G-shock sensor was not = sensor’ incorporated into the RF 
mature enough to be assessed during the demonstrations because of | tag._It gave an indicator if there 
inadequate electronic coupling between the G-shock sensor andthe — was a problem with ammuni- 

RF tag board, which caused false shock detections. As aresult.the tions.” 

G-shock sensor was not assessed during the demonstrations. Warfighter comment 


For two days during the OCONUS, CONUS. and maritime demon- 

strations, the assessment team recorded hourly temperature and RH 

measures using a Kestrel 4000 weather monitoring station. The temperature and RH mea- 
surements were compared with the environmental histogram from the RF tag and results 
indicated that temperature and RH measures correlated with data captured by the assessment 
team. The RF tags collected and recorded environmental data on the tagged munitions pallets 
every 60 munutes during all demonstrations. 

When the RF tags were created, the environmental sensors began collecting hourly tempera- 
ture and RH measures. Most of these RF tags had more than 4700 hours of continuous 
operation from March to August 2004. The assessment team did not observe any failures in 
the temperature and RH sensors. The TECEVAL data revealed the RF tag can record and 
store environmental data for 45 days (1080 data points). 


Dunng the demonstrations, the TM randomly selected the upper and lower user-selectable 

and RH limits. Many of these settings were not operationally representative, but 
did allow assessment of limit flag reporting (1.e., when temperature reached the limit, a flag 
was set and data were reported to the PP). 


The warfighters provided comments about the RF tags’ environmental sensors. The 

warfighters indicated that the temperature sensor capability would provide valuable informa- 

tion. A Miesau warfighter stated. “The remperature [user-selectable limit] was easy to 

change” and will be especially useful when the temperature range requirements for critical 

munitions are defined in the future. 

Although the RF tag performed well during the demonstration, some of the warfighters 

expressed concems about the cost of the RF tag, since they were not designed to be reusable. 

There were also several discussions about the RF tag environmental sensors. One warfighter 

stated “The [RF] tags on the outside of container dont accurately give temperature and 

RH inside the munitions pallets.” Another warfighter wanted to know immediately when 

environmental flags were tnpped. In addition, one SME commented that it would be usefill to 

know exactly when an environmental flag occurred, and requested that a date/time stamp be 

added. A CAAA warfighter stated that the RH data would not provide useful information. 
EEE ——— 
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Warfighters indicated that having a RF tag that shows when a muuitions pallet was dropped is 
only beneficial if the RF tag can accurately record the distance dropped or shock received. 
Finally, as stated earlier im the report, the G-shock sensor was not demonstrated during the 
MUA. 


Munitions Management 


Pallet Receipt 

Pallet receipt consisted of receiving incoming tagged mumuitions pallets. creating and attaching the 
RF tag to the mumitions pallets, moving the tagged mumitions pallets to an mutual storage location, and 
adding the munitions pallets and their content to the inventory. Dunng the demonstrations, the 
warfighters were able to create new RF tags using the HHR. The warfighters stated “the creation 
of a RF tag was easy and straightforward” however, the assessment team observed some 
problems with HHR reading asset data from the RF tags. It may require the warfighter to move 
the HHR to different angles to read the RF tags. During the demonstrations, HHR took two to five 
nunutes to read and write to a RF tag, and many warfighters complained about the slow processing 
speed. 
Dumng the demonstrations, the warfighters were shown two methods to attach the RF tags to the 
nuumitions pallets. The first attachment method used wood screws and the second attachment 
method used a special double-sided adhesive tape. At the Miesau and CAAA demonstrations. the 
TM used four RF tags and tag holders to evaluate adhesive attachment to metal and wooden 
nuumitions pallets. Tag holders remained attached to both metal and wooden munitions pallets during 
the demonstrations, but questionnaire results indicated that the RF tags adhered better to metal 
pallets than to wooden pallets. Warfighters were concemed that the tag holder would fall off the 
wooden pallets and recommended securing them with wood screws. 
Dunng the maritime demonstration, the CAAA logistics personnel attached 32 RF tags to the 
bottom of the mumutions pallets for siupment to Norfolk Naval Station, VA via commercial ground 
transportation (Figure 8). Thirty-two RF tags were placed on the nmmitions pallets at CAAA IN 
and shipped by commercial ground to the Norfolk Naval Station, VA. Because Navy nmmitions 
personnel use chains to cradle the pallets from below during loading of the nmmutions pallets onboard 
ship. alata nad rman ni cone ie ern pasar 
tags during loading. Thirty-one of 32 RF tags 
were removed from the mumitions pallets and 

; were undamaged and functional. The HHR 
— was used to locate the mussing RF tag. which 

; was still attached to a mumitions pallet when it 
was moved to the USS Harry S. Truman’s 
magazine. The RF tag was fimctional but not 
visible. The assessment team was not able to 
venfy if the TM staff was able to remeve the 
missing RF tag. 






RF tags and tag 
holders were 
placed here 


Figure 8. RF Tag Placement During Transport to the USS 
Harry S. Truman: Thirty-rwo RF tags were placed on the 
munitions pallets at CAAA, IN and shipped by commercial 
ground to the Norfolk Naval Station, VA. 
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After RF tags were placed on the munitions pallets, the pallets were moved to a magazine to 
be interrogated by the RCU. The RCU and RFE were effective in detecting 200 RF tags 
inside the earth-covered magazines, storage warehouse, and ship magazine. At the OCONUS 
demonstration, the RCU detected 199 out of 200 RF tags (99.5%) in the earth-covered 
magazine, and 169 out of 174 RF tags (97.1%) in the ASP. In the CONUS demonstration, the 
RCU detected 197 out of 200 RF tags in the earth-covered magazine. In both demonstrations, 
the first interrogation cycle was 14 minutes on the short read and two hours and 30 minutes on 
the long read. During the second and third interrogation cycles, ATOS was able to detect 
100% of the RF tags. At the maritime demonstration, the RCU was able to detect 200 RF 
tags on the USS Harry S. Truman on the first interrogation cycle. Questionnaire results 
indicated the RF antenna coverage was acceptable. A WLAN or LAN was used to transfer 
the interrogated RF tag data (e.g.. missing RF tag, environmental flags. and updated environ- 
mental history data) from the RCU to the PP. 


When asset information was loaded on the RF tag. the HHR created 


and stored a transaction record for each RF tag. The transaction RF oa — is muuch 
record contains the asset information and the latest update in the seed : = hp — 
environmental data. During the CAAA demonstration, the assess- ocala 

ment team observed that when the warfighters docked the HHR to Warfighter comment 


the PP, some transaction records did not get transferred to the PP 

and other transaction records were deleted from the HHR. During the maritime demonstra- 
tion, a warfighter attempted to transfer 180 transaction records from the HHR to the PP. The 
HHR software timed out after a few nunutes during data transfer to the PP. The TM devel- 
oped a workaround solution to demonstrate the pallet receipt capability by reducing the 
number of transaction records (50 records per packet) being transmitted from the HHR. The 
assessment team also observed an interoperability issue that occurred when two HHRs were 
within 20 feet of each other, causing the HHRs to lock up. Despite these issues, questionnaire 
results indicated that the warfighters believed the HHR was user friendly and easy to use 
during pallet receipt and that the stylus was extremely useful in data entry even when they 
were wearing bulky work gloves. Questionnaire responses also indicated that warfighters 
believed the PP was easy to use for pallet receipt. 


Inventory Maintenance 


Dunng the demonstrations, the RCU maintained the mventory through interrogations of the RF 

tags every 60 minutes. Once the inventory list reached a steady-state condition, the RCU only 

Teported exceptions to the inventory. Ifthe RCU encountered any environmental alarm flags. low 
battery power flags, or missing or new RF tags. a wamung indicator was graphically displayed on 

the PP computer screen. These warming indicators remained on the computer screen until the 

ATOS operator took action to resolve the problem or cleared the wamung indicators. 

When conducting an inventory using an HHR, warfighters downloaded the inventory list to the 

HHR from the PP and performed a roll call fiction (query each RF tag) on the HHR. The HHR 
compared the downloaded inventory list to the RF tags located inside the magazine. The 

warfighters were able to quickly inventory 60 RF tags 

inside the USS Ponce’s magazine and identify new or “[ATOS will] reduce the time needed 
missing tagged numuitions pallets using the HHR. The to conduct an inventory of a 60 x 40 x 
assessment team compared the count of the handwnitten 15 foot earth-covered magazine from 
inventory list, the HHR transaction records, and the count —_—six to eight hours to an hour or less.” 
on the PP inventory. After the inventory was completed. Warfighter comment 
the HHR was docked to the PP for the PP to receive the 
updated inventory from the HHR. 
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The assessment team observed that the RCU successfully monitored incoming and outgoing 
tagged munitions pallets during the Miesau and CAAA demonstrations. The warfighters were 
able to locate RF tags, verify the inventory list, and identify new and missing RF tags inside 
the magazine. Assessment team observations indicated that the PP had sufficient bandwidth, 
connectivity, and data storage to perform the inventory maintenance task. Questionnaire 
results indicated that the warfighters believed that the RF tags, HHR, and PP were easy to 
configure for inventory maintenance. 

Duning the maritime demonstration, the PP locked up when downloading the inventory of 200 
RF tags to the HHR. The TM identified this as a PP software problem. In addition. the 
assessment team observed that because RCU operation could not be paused during its 
interrogation cycle, the warfighters were unable to conduct an inventory until the interrogation 
cycle was finished. 


Pailet Movement 


Dunng pallet movement, warfighters used the HHR to update the location field on the RF tag, 
and the RCU updated the location field on the PP inventory list. During the pallet movement 
demonstration, no RF tags were physically attached to the munitions pallets stored inside the 
magazine. The assessment team selected five RF tags and had them removed from an earth- 
covered magazine at Miesau and CAAA demonstrations. Two RF tags were moved to a 
storage warehouse that had a different RCU. The other three RF tags were placed inside a 
munitions container and moved away from the magazine to simulate munitions pallets in 
transit. The RCU detected the two RF tags inside the storage warehouse as existing inventory 
and updated the RF tag location field, and the warfighter used the HHR to update the RF tag 
information. The PP identified the three missing RF tags by ID number. When the three RF 
tags were placed back inside the magazine, the missing RF tag notifications were removed 
from the PP display during the next RCU interrogation cycle. Dunng the maritime demonstra- 
tion, the assessment team removed 20 RF tags from the ship’s magazine to simulate a pallet 
movement. The PP again identified the missing RF tags by ID number. When 10 RF tags 
were placed back into the magazine, the PP identified only the ten remaining missing RF tags. 
As long as the tagged munitions pallets remained with the assigned RCU ID, they could be 
moved anywhere within the magazine without the PP indicating nussing RF tags. Ifthe 
tagged munitions pallets were removed from the assigned RCU area of coverage. the PP 
notified the warfighters within 60 minutes of the last RCU interrogation cycle that the pallets 
had been moved. If the tagged numutions pallets were moved to a different building with a 
different RCU ID, the RCU would update the RF tag with a new RCU ID number. The 
assessment team verified the location changes in the HHR and PP. If the tagged mumitions 
pallet was moved before the start of the next interrogation cycle, the PP does not receive a 
flag waming. The assessment team observed that the PP, HHR, and RCU had sufficient 
bandwidth, connectivity, and data storage to perform pallet movement. Questiomnaire results 
indicated that the warfighters believed the HHR and PP were easy to configure and use for 
pallet movement. 


Pallet Issue 
Duming pallet issue, the warfighters simulated the issue of one or more individual munitions 
items from tagged munitions pallets, using the HHR to update the munitions count on the RF 
tag. The HHR was later docked with the PP and the inventory was updated to reflect the 
new munitions count. The RCU interrogated the RF tags every 60 minutes to collect any 
changes in the asset data. Questionnaire responses indicated the warfighters thought the 
HHR was easy to configure and use to update the munitions count. The assessment team 
observed that the HHR had sufficient data storage for pallet issue. 
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Pallet Transfer 


Dunng pallet transfer, warfighters prepared the tagged muuutions pallets for shipment outside the 
depot, using the HHR to download the latest asset and environment data and any environmental 
flags. The warfighters docked the HHR to transfer the data to the PP. The RF tag ID number 
was removed from the inventory and the data archived. The assessment team observed the 
warfighters download the asset and environmental data using the HHR. The assessment team 
also observed warfighters using the HHR keypad and stylus to update the asset data and transfer 
the information to the PP. The PP, HHR, and RCU had sufficient data storage for pallet transfer. 
Questionnaire results indicated that wardighters believed the HHR to be instrumental in transfer- 
Ting tagged munitions pallets. 

Munitions QA 


The Munitions QA task was not assessed during the demonstrations (see limitations and con- 
stramts section). From a Quality Assurance Specialist Ammunition Surveillance specialist wew- 
point, the current design does not have any applicability for quality control. Questionnaire re- 
sponses indicated the RF tag does not have any fields that contain information about the inspection 
history, recent suspense information, or condition code changes that could impact serviceability of 
the nuuutions items. 


System Characterization 


Training 
The warfighters indicated the training was effective in providing the necessary information to 
operate the ATOS system dunng the demonstrations. The taming matenal was easy to understand 
and the system was easy to operate. All of the warfighters wanted more 
hands-on time with the system dunng the classroom instruction. They “For myself, the hands-on training 
also would like to have had a training/user manual available during the was how I really began to under- 
classroom instucnon. stand how the units work.” 
Component Ruggedness seer same oom 
The assessment of the ATOS component ruggedness was limited. The 
TECEVAL data revealed that the RF tags meet the ruggedness requirements stipulated im the 
FRD. The assessment team collected a limited amount of handling data during shipment from the 
CAAA to Norfolk Naval Station. All RF tags performed nominally during the demonstrations. 
The assessment team observed two moperable RF tags (one RF tag from the Miesau demonstra- 
tion and one RF tag in the maritime demonstration). The TM is currently investigating the causes 
of the failures. The assessment team did not observe any HHR or PP failures. The assessment 
team observed two RCU failures. The first failure occurred at the Miesau demonstration and was 
caused by a faulty flash memory card. The second failure occurred at the maritime demonstration 
and was caused by a faulty power supply. 


Power Requirement 

No RF tags were replaced during the demonstrations due to low battery power. Question- 

naire results indicated that the warfighters would prefer to be able to replace RF tag batteries 

rather than throwing away the entire RF tag. The HHR used two rechargeable batteries and 

docking stations to ensure power was maintained. The RCU was able to operate as long as 

24 VDC power was available. Dung all demonstrations, a portable gas electric generator, 

High Mobility Multipurpose Wheeled Vehicle battery, or 120/220 volts alternating current 

(VAC) line power and 24 VDC step down were used to provide electrical power. The 
a __________. 
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warfighters stated that the PP power requirements were straightforward and easy to accom- 
modate as long as the PP was hosted in a facility that had 120/220 VAC electrical power. 


Installation 


During the ATOS ACTD, the TMs staff installed the RCU, RFEs, WLAN or LAN inside the 
magazines, storage warehouse, ASP. and ship magazines. The number of RFEs required to 
detect 200 RF tags was determined by the size and configuration of the magazine. Typically, 
the installation took approximately two days to accomplish at each demonstration site. On the 
third day, the TMs staff would lay out 200 RF tags and conduct several experiments to 
determine the optimum position for the RFEs to detect 100% of RF tags inside the magazine. 
Questionnaire results indicated the warfighters believed the location and accessibility of the 
RCU were acceptable for the demonstrations. Tie wraps were used to install the RFEs, while 
the RCUs were mounted on portable stands. The PP installation was straightforward since the 
software was pre-installed on a laptop computer. 


Security 


The network security of the ATOS system was not assessed due to the unavailability of 
information security SMEs. 
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CONCLUSIONS AND RECOMMENDATIONS 


The ATOS system demonstrated potential nulitary utility and was effective in providing 
environmental surveillance and enhancing mumitions management. Although there were some 
technical problems encountered during the demonstrations, the system was able to provide the 
warfighters near real-time environmental data and support five out of the six munitions 
management tasks during the demonstrations. Table 7 identifies the environmental surveil- 
lance and munitions management ratings based on the assessment team’s findings. 
Table 7. MUA Unility Ratings: The system was effective in providing the 
warfighter with environmental surveillance data and enhancing munitions 
management. 
ATOS System Capability Rating 
Environmental Surveillance 


Munitions Management 
Pallet Receipt 





> 


E 
| 
QPPer! epipp<« 


Legend 

A Demonstrated utility, deployable now; minorimprovements recommended. 
A Potenfal utility, minor improvements recommended. 

Ny’ Potential utility; majorimprovements required. 

Vv No utility demonstrated; possible utility; complete redesign needed, 


Environmental Surveillance 


The RF tag demonstrated military utility and was effective in providing the warfighters a new 
capability for collecting environmental data on tagged munitions pallets. RF tag histogram 
data correlated closely with the measurements taken by the assessment team during the 
demonstrations. The warfighters would like to have the sensors embedded in the muuitions so 
that accurate environmental readings can be obtained for individual items and for conditions 


inside the pallet. 
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Munitions Management 


The consensus of the warfighters was that the ATOS system demonstrated mulitary utility and 
enhanced the logistics process for managing munitions pallets. The warfighters were im- 
pressed with the automated capability to tack and monitor incoming and outgoing tagged 
niunitions pallets during pallet movement, pallet issue, and pallet transfer. Listed below are 
the conclusions for each munitions management task. 


Pallet Receipt 
The ATOS system demonstrated potential military utility. but will require major improvement. 
The system was effective in performing pallet receipt of 20 to 60 RF tags and reduced the 
time to receipt incoming muuutions pallets. However, the technical problems encountered 
during demonstrations make the system performance unacceptable. Reading and wniting data 
to and from the RF tag using the HHR was time consuming, there were data transfer prob- 
lems between the HHR and PP. and an interoperability problem occured when two HHRs 
were within 20 feet of one another. This system performance was unacceptable to the 
wartfighter since a depot routinely receives large numbers of incoming munitions pallets. 


Inventory Maintenance 


The ATOS system demonstrated potential military utility. but will require major improvement. 
The system was effective in performing the inventory task and reduced the time needed to 
conduct an inventory. “The HHR and RCU were usefil in tracking incoming and outgo- 
ing tagged munitions pallets and identifying alarms.” However, the PP was unable to 
download the entire inventory of 200 RF tags without locking up. This fimction must be fixed 
before the system is deployed to the warfighter. 


Pallet Movement 
The ATOS system demonstrated potential military utility, but will require minor improvement. 
The system was effective in performing pallet movement, with the warfighters being able to 
use the HHR to update the location field on the RF tag. However, the system does not alert 
the PP with a flag indicator when a RF tag has been moved to a different building and interro- 
gated by a different RCU. A wamung indicator should be incorporated in the PP software 
which will identify any tagged munition pallets that were moved before the next interrogation 
cycle. Otherwise, the PP was able to verify that tagged nmmitions pallets had been moved 
from one location to another within the same depot. 


Pallet Issue 
The ATOS system demonstrated mulitary utility and was effective in performing pallet issue, 
with warfighters being able to use the HHR to update the munitions count on the RF tag and 
the RCU successfully captunng the munitions count changes and reconciling them with the 
inventory on the PP. 


Pailet Transfer 


The ATOS system demonstrated military utility and was effective in performing the pallet 
transfer. The warfighters were able to download asset and environmental data from the RF 
tags using the HHR and transfer the data to the PP where the RF tag ID was removed from 
the inventory and the data archived. 


= —————— 
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Munitions QA 
The Munitions QA task was not assessed by the assessment team during the demonstrations. 


System Characterization 


Training 
The overall feedback the assessment team received from the warfighter was that the ATOS 
system training was adequate and provided the necessary information to operate the system. 
The warfighter indicated the traming was effective in providing the necessary information to 
operate the ATOS system during the demonstrations. 

Component Ruggedness 
The ATOS system components performed nomunally during the demonstration. The assess- 
ment team did not observe any major failures that the TM could not recover in a couple of 
hours. ATOS performed most of the munitions tasks with no major issues. 
Power Requirement 
No major issues were identified at the demonstration sites in providing the necessary power to 
support ATOS. The warfighters wanted a capability to replace the batteries in the RF tag. 
Installation 

ATOS component installation was straightforward and easy. No major issues were identified 
at the demonstrations. 

Security 
Security was not assessed by the assessment team during the demonstrations. 


Recommendations 
Recommendations are based on warfighter inputs and the results from the ATOS MUA. 
These recommendations were separated into two categories: 11 recommendations for system 
improvements and two recommendations for firther testing. 
Environmental Surveillance 
* Provide time and date stamp association to determine when a critical munitions item 
experienced extreme temperature and RH vanations. 

* Develop reusable RF tags 

* Develop and demonstrate a G-shock sensor capability 

* Attach RF tags to inside of individual munitions containers 

* Develop environmental sensors that are embedded in the munitions 


EE, 
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Munitions Management 

+ Reduce HHR read and write processing speed from 5 minutes to 40 seconds or less 

+ Fix the data transfer problem with the HHR so that it is able to transfer the designed 
number of transaction records (approximately 1,000) to the PP 

* Develop procedures or software modifications allowing the use of HHRs in close 
proximity of each other 

* Develop a pause function on the RCU to allow the warfighters to perform logistical 
tasks using the HHR 


* Fix the PP data transfer problem so that the HHR can download the entire inventory 
list to the HHR 


* Modify the RF tag so that munitions QA can document condition codes and inspection 
history 
Test Refinement 
+ Evaluate the interoperability of the Service Ammumition AIS and EDB with ATOS 
* Evaluate network security of the ATOS system 


Se 
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ACTD advanced concept technology demonstration 
AFOTEC Air Force Operational Test and Evaluation Center 


AIS automated information system 

AIT automatic identification technologies 

AOR area of responsibility 

ASP ammunition supply point 

ATOS Advanced Technology Ordnance Surveillance 
CAAA Crane Army Ammunition Activity 

col critical operational issues 


CONOPS concept of operations 
CONUS continental United States 


COTS commercial off-the-shelf 

Det 1 Detachment 1 

DoD Department of Defense 

DODIC DoD Identification Code 

EDB environmental database 

F fahrenheit 

FRD functional requirement document 
G-shock gravity-shock 

HERO Hazards of Electromagnetic Radiation to Ordnance 
HHR handheld reader 

ID identification 

JMC Joint Munitions Command 

LAN local area network 

LCD liquid crystal display 

MB megabytes 

MEMS micro-electromechanical systems 


MHz MegaHertz 


MOE measures of effectiveness 


MOP measure of performance 

MUA military utility assessment 

mW nulhiwatt 

NAVSEA Naval Sea Systems Command 
NSN National Stock Number 

NSWC Naval Surface Warfare Center 
OCONUS outside the continental United States 
OM Operational Manager 

PP preprocessor 

QA Quality Assurance 
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RCU reader control unit 


RF tadio frequency 

RFE radio frequency extenders 
RH relative humidity 

SME subject matter expert 
TECEVAL ___ technical evaluation 

™ technical manager 
USEUCOM _ US European Command 
USS United States Ship 

VAC volts alternating current 
VDC volts direct current 
WLAN wireless local area network 
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Table B-1 correlates MOEs addressed during the MUA with environmental surveillance and munitions manage- 
ment tasks. 


Table B-1. Correlation of MOEs to Environmental Surveillance and Munitions Management Tasks: 
The MOEs were correlated to the environmental surveillance and munitions management tasks to 
illustrate the assessment methodology. 


Correlation of Munitions Management and Environmental Surveillance to MOEs 


Munitions Management 


1.1 Capability of ATOS system to provide 
container location data 

12 Capability of ATOS system to accurately 
report and maintain environmental data on 


1.3 Capability of ATOS RF tags to 


+4 Capabity of ATOS HHR and RCU b 

communicate and transfer data to PP 

15 Capability of ATOS PP to interoperate 

with Service Ammunition AlS 

1.6 Capability of ATOS PP to interoperate 

with EDB 

1.7 Characterization of network bandwidth 

connectivity to ATOS PP 

tle oe Momo protectdata| 
& os amto pro d 

sufficient data s ge 

1.10 ares of ATOS RCU antenna 








2.8 Supportability of ATOS installation 
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Table C-1 describes the 19 scenarios that were used to demonstrate environmental surveillance and munitions 
management capabilities. Each scenario was repeated at least five times by the participants to ensure they 
completely understood the ATOS system operations. 
Table C-1. MUA Scenario: Nineteen scenarios were conducted to ensure all ATOS functions were 
assessed. 


ATOS Scenario 





Scenario Script Description 
1. Demonstrate capability to conduct the Operators perform administrative and setup functions with HHR 
administrator funcilons of the PP and HHR. | prior to start of the demonstration, which includes docking the HHR 
to the PP and configuring the PP and the RCU. 

. Demonstrate PP operations in conjunction | Operators conduct munitions management tasks using the HHR, 
with the HHR and RCU operations. RCU, and PP. 
3. Demonstrate how to set the sensor alarm | Operators set RF tag sensor alarm thresholds during initial creation 
thresholds. of RF tags; thresholds for some RF tags are set above or below 
ambient temperatures to trigger alarms. 
ee tees ee Operators create 25 RF tags using the HHR keypad to enter asset 
keyp information. 
5. Create new RF lags using data from the Operators create five RF tags using asset information fom the PP 
j 











PP. downloaded from the PP to the HHR and transferred to RF tag 

6. Create new RF tags using data from bar | Operators create five RF tags using data from bar code labels. 

code labels. 

7. Perform inner-area of movement tagged | Operators move RF tags from one location to another within the 
munitions pallets. munitions faci it 

_8. Move tagged munitions pallet with abad | Operators move bad RF tags from one location to another within the 
| RF tag (not responding dus to dead munitions facility and replace bad RF tags with fully functional items. 
battery). 

9. Move tagged munitions pallet with a bad 
RF tag (reason other than dead battery). 


Operators move bad RF tags from one location to another within the 
munitons facility, simulate dead RF tag, and simulated bad RF tag 
with fully functional item. 

10. Movement of asset with a tripped sensor | Operators perform inventory actions to address tripped sensor 
alarm. alarms. 

11. Movement of tagged munitions pallet Operators move an RF tag with low battery from one location to 
with low battery. another within the munitions facility and replace the RF tag with one 
that has a fully functional batt 

Operators edit asset information on RF tags using the HHR, read 
the edited information using the RCU, and update the PP using the 
HHR via docking station. 

13. Asset receipt Operators perform asset transactions, such as pallet receipt. 




















12. Change RF tag data. 






14. Asset issue (entire pallet with RF tag). | Operators use the HHR to issue contents of an entire tagged 
munitions pallet 

15. Asset issue (partial pallet - RF tag Operators use the HHR to issue some of the contents from a tagged 

remains). munitions pallet, noting the quantity and type of items issued. 


| 16. Asset transfer (shipment). Operators transfer tagged munitions pallet fom one location to 
another and validaie transaction (Le., determine if asset data 
elements in the PP match those that were t ad | 
17. Perform inventory of all tagged munitions Operators use the HHR roll call function to conduct munitions | 
pallets in a storage location. invento | 
18. Search for and identify various RF tag | Operators use the HHR roll call function to find tagged munitions 
using HHR find function. pallets. 
19. Download RCU files. Operators use the HHR to download interrogation files from the 

RCU. 
 ——————————————_ 
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This appendix presents the detailed results for the MOEs and MOPs used to derive the results and conclusions 
for the ATOS MUA. The bar charts show aggregated responses from operators, logistics, and munitions partici- 
pants. The analysis revealed no significant differences in warfighters’ opinions at the demonstrations. 


MOE 1.1 Capability of ATOS system to provide container location 
data 


This MOE assesses the ATOS capability to provide and update location data (building grid coordinate) on tagged 
munitions pallets. During the demonstrations, the RF tag was able to maintain location field data and gave the 
warfighter a capability to update RF tag location information. 


MOP 1.1.1 Capability of ATOS system to provide container location data 


The HHR was used by the warfighters to mput building coordination grids into the RF tag location data field and 
the RCU then updated the RCU location field on the PP. At each demonstration site, the warfighters were tasked 
to change the location field on 20 RF tags. The assessment team verified the location changes in the HHR and 
PP. As long as the tagged mumitions pallets remained within the assigned RCU ID, the tagged munitions pallets 
could be moved anywhere in the magazine and the PP would not flag a missing RF tag. If the tagged munitions 
pallets were moved to a different building with a different RCU ID before the 60-mumute interrogation cycle, the 
new RCU would update the RF tag with a new RCU ID number. The warfighter would not be notified when 
tagged mumutions pallets were moved to a different location since the PP would reconcile the inventory and show 
the tagged munitions pallets were picked up by another RCU. Therefore. the warfighter would not know the 
location of a moved tagged munition pallet without referring to the PP. One warfighter stated the RF tag location 
capability is “Much easier than the current method used now.” 





MOP 1.1.2 Operator assessment of RF tag location capability 


The questionnaire responses indicated the RF tag location capability was acceptable in supporting the warfighter 
in locating tagged munitions pallets (Figure D-1). The warfighters believed the RF tag location capability made 
their job of locating tagged munitions pallets easier. A CAAA warfighter stated. “The accuracy of the location 
reports would be useful. However, need to see ATOS work in a jam stow [munitions pallets stacked end-to- 
end] environment.” 


Rate the acceptability of RF tag location capability 


a 


oy 


Number of Respo nses 





Figure D-1. Questionnaire Results, RF Tag Location Capability: 30 of 31 warfighters agreed 
that the RF tag location capability was acceptable. 
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MOE 1.2 Capability of ATOS system to accurately report and 
maintain environmental data on containers 

This MOE assesses RF tag capability to accurately measure temperature, RH, and G-shock: and to alert the 
operator if an environmental condition exceeds the threshold of the munitions. Although the RF tags collected 
environmental data, the assessment team received a few negative comments. One CAAA warfighter felt the 
RH data would not provide useful information. Another warfighter stated, “Tags on the outside of container 
does not [sic] accurately give temperature and RH inside the munitions pallets.” 

MOP 1.2.1 Percentage of ATOS RF tags that accurately report and maintain 
environmental data of the containers 

Dunng TECEVAL, 20 RF tags were placed inside an environmental chamber to determine if the RF tags accu- 
rately recorded the temperature and RH. The histogram data revealed all 20 RF tags (100%) accurately re- 
corded the temperature and RH data. Dunng the demonstrations, the RF tag histogram data correlated with the 
temperature and RH data collected by the assessment team. 

MOP 1.2.2 RF tag temperature accuracy 


The TECEVAL data indicated the temperature sensors on the 20 RF tags were accurate within + 1°F readings 
from the environmental chamber settings. The assessment team was unable to measure temperature accuracy 
during the demonstrations because the method on how the RF tag collects temperature data was changed to 
develop the histogram data. Therefore, the data was unavailable to the assessment team. 

MOP 1.2.3 Reliability of RF tag temperature sensor 


When the RF tags were initialized, they continued to collect hourly temperature measures. Most of the histogram 
information RF tags had more than 4.700 hours of continuous operation. The assessment team did not observe 
any failures in the temperature sensor. 

MOP 1.2.4 RF tag relative humidity accuracy 

The TECEVAL data indicated the RH was accurate within = 10% of the environmental chamber settings. 
TECEVAL data also indicated that the RH sensor would give erroneous readings if the RH was above 95% or 
below 10%. The assessment team was unable to measure RH accuracy during the demonstration because the 
method on how the RF tag collect RH data was changed to develop histogram data. 

MOP 1.2.5 Reliability of RF tag relative humidity sensor 

When the RF tags were initialized, they continued to collect hourly RH measures. Most of the RF tags were 
initialized in March 2004 (more than 4.700 hours). The assessment team did not observe any failures in the RH 
sensor. 

MOP 1.2.6 RF Tag gravity-shock accuracy 

Not assessed. The G-shock sensor was not available to support the demonstrations. 

MOP 1.2.7 Reliability of RF tag gravity-shock sensor 

Not assessed. The G-shock sensor was not available to support the demonstrations. 

MOP 1.2.8 Number of days RF tag accurately records and stores 
environmental data 


Not assessed. The TECEVAL data revealed the RF tag could record and store 45 days (1080 data points) of 
environmental data. Since RF tags were cleared for each demonstration, the assessment team was not able to 
assess maximum number of days the RF tag could accurately record and store environmental data. The assess- 
ment team observed no problems with the RF tags holding all the environmental data during the demonstration. 
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MOP 1.2.9 Frequency of temperature reports stored in RF tag 

The RF tag was designed to collect temperature reports every 60 minutes when initialized. The results indicated 
the RF tag collected and recorded temperature data every 60 minutes. 

MOP 1.2.10 Frequency of relative humidity reports stored in RF tag 

The RF tag was designed to collect RH reports every 60 munutes when initialized. The results indicated the RF 
tag collected and recorded RH data every 60 minutes. 

MOP 1.2.11 Number of temperature reports above a user-selectable limit 
TECEVAL data showed 20 RF tags had their upper temperature limits set. All 20 RF tags had their environment 
flags tipped at the appropriate temperature settings. 

MOP 1.2.12 Number of temperature reports below a user-selectable limit 
TECEVAL data showed 20 RF tags had their lower temperature limits set. All 20 RF tags had their environment 
flags tipped at the appropriate temperature settings. 

MOP 1.2.13 Number of relative humidity reports above a user-selectable 
limit 

TECEVAL data showed 20 RF tags had their upper RH limits set. All 20 RF tags had their environment flags 
tipped at the appropriate RH settings. 

MOP 1.2.14 Number of relative humidity reports below a user-selectable 
limit 

TECEVAL data showed 20 RF tags had their lower RH limits set. All 20 RF tags had their environment flags 
tripped at the appropriate RH settings. 

MOP 1.2.15 Number of gravity-shock sensor reports above the first limit on 
the x-axis 

Not assessed 

MOP 1.2.16 Number of gravity-shock sensor reports above the second limit 
on the x-axis 

Not assessed 

MOP 1.2.17 Number of gravity-shock sensor reports above the first limit on 
the y-axis 

Not assessed 

MOP 1.2.18 Number of gravity-shock sensor reports above the second limit 
on the y-axis 

Not assessed 

MOP 1.2.19 Number of gravity-shock sensor reports above the first limit on 
the z-axis 

Not assessed 

MOP 1.2.20 Number of gravity-shock sensor reports above the second limit 
on the z-axis 


Not assessed 
——————————————————————————_—_—_—_—_—_—_—_—————— 
Advanced Technology Ordnance Surveillance Final Report D-5 


For Official Use Only 


125 


MOP 1.2.21 SME assessment of RF tag temperature sensor 


SMEs and operators indicated that the RF tag temperature sensor was acceptable for measuring tagged mumi- 
tions pallets ambient temperature (Figure D-2). One munitions SME provided this comment, “Although the 
temperature sensor and its application are acceptable, the ammunition surveillance community has not 
caught up to this level of applicability.” One ATOS operator stated that it was “Nice to see that this capabil- 
ity was incorporated into the tag. It gave the warfighter an indicator if there was a problem with 
ammunitons.” SMEs also indicated that although the RF tags on the outside of the container do not provide 


temperature readings for the inside of the container, the capability was better than not having any temperature 
measurements. 


Rate the acceptability of RF tag temperature sensor 


C et \ t Jy YF 





Figure D-2. Questionnaire Results, RF Tag Temperature Sensor: All respondents agreed that the 
temperature sensors were acceptable. 
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MOP 1.2.22 SME assessment of RF tag relative humidity sensor 
SMEs indicated the RH sensor concept was good (Figure D-3); however, the cwrent application of the RH 
sensor needs to be changed to be usable to the naval warfighters. The placement of the RH sensor should reflect 
what is going on inside of the sealed container. not on the outside of the container. One warfighter indicated “The 
need to get relative humidity sensor inside the container.” The questionnaire responses indicated that the RH 


sensor was acceptable for providing RH measures on tagged numitions pallets. 


Rate the acceptability of RF tag relative humidity sensor 
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Figure D-3. Questionnaire Results, RF Tag RH Sensor: 23 af 25 respondents agreed that the RH 
sensor was acceptable. 
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MOP 1.2.23 SME assessment of RF tag environmental sensor reliability 

The SME indicated the RF tag environmental sensor was reliable based on the limited amount of time the 
warfighters used the RF tags (Figure D-4). However. all warfighters indicated the RF tags performed well 
during the demonstrations. 


Rate the reliability of RF tag environmental sensor 





Figure D-4. Questionnaire Results, Tag Environmental Sensor: 100% of warfighters agreed that 
the RF tag environmental sensor was reliable during the demonstration. 


MOP 1.2.24 SME assessment of RF tag gravity-shock sensor 

Not assessed 

MOE 1.3 Capability of ATOS RF tags to communicate with RCU 
or HHR 


This MOE assesses the RCU capability to interrogate the RF tags and the HHR capability to query RF tags for 
asset and environmental data. The questionnaire results indicated the data transfer speed for reading RF tag data 
using the HHR was too slow while conducting pallet receipt. During pallet issue, the RF tags were able to 
communicate the pallet content changes to the RCU and HHR. Duning pallet transfer, RF tags were able to 
communicate the latest asset and environmental data to the RCU and HHR. 


MOP 1.3.1 Number of ATOS RF tags that were able to communicate with 
RCU or HHR 

Miesau demonstration: During the first interrogation cycle, the RCU detected 199 out of 200 RF tags (99.5%) in 
a 60 x 40 x 15 foot earth-covered magazine using three RFEs. The RCU detected 169 out of 174 tags (97.1%) 


in a 70 x 200 foot ASP using eight RFEs. Duning the second interrogation cycle, 100% of the RF tags were 
detected. 
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CAAA demonstration: The RCU detected 197 out of 200 tags (98.5%) in both the 100 x 50 x 10 foot earth- 
covered magazine using 6 RFEs and the 200 x 50 foot storage warehouse using 12 RFEs. Dunng the second 
interrogation cycle, 100% of the RF tags were detected. 


Mantime demonstration: The HHR detected 60 out of 60 RF tags (100%) ina 10 x 22 x 8 foot magazine on the 
USS Ponce. On the USS Harry S. Truman, the RCU detected 200 out of 200 RF tags (100%) ina 45 x 80x 10 
foot magazine using 3 RFEs. 

MOP 1.3.2 Automatic reporting frequency from RF tag to RCU 

ARCU performance log was used at the Miesau and CAAA demonstrations to monitor the start and stop times 
of the RCU interrogation cycles. The results indicated that the RCU interrogated the RF tags every 60 minutes. 


MOP 1.3.3 On-command reporting to HHR 


On-command reporting 1s the HHR capability to perform roll call or find functions. At the Miseau demonstration, 
the warfighters used the HHR find fiction to locate five tagged munitions pallets. All five tagged pallets were 
located using the HHR_ No major problems were observed during the demonstrations. At the CAAA demon- 
stration. warfighters were able to conduct an inventory of 20 RF tags im an earth-covered magazine within 20 
minutes. The warfighters indicated this fumction was a time saver. During the maritime demonstration, the HHR 
was effective in performing a rapid tally (within 10 minutes) of 60 RF tags inside the USS Ponce’s 10x 22x 8 
storage facility. On the USS Harry S. Truman, Det 1 AFOTEC performed a roll call of 200 RF tags within 15 
minutes. “The warfighters indicated the system seemed slow, but I feel it will execute better when the bugs 
are worked out.” Figure D-5 shows responses to the on-command reporting to the HHR questiomnaire state- 
ment. 


Rate the acceptability of on-command reporting to HHR 
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Figure D-5. Questionnaire Results, On-Command Report: All responses agreed that the HHR on- 
command reporting was acceptable 
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MOP 1.3.4 On-command memory reset from HHR 


Duming the creation of a new RF tag. the RF tag was given a command by the HHR to clear its memory. The 
assessment team observed that 18 warfighters successfully cleared the RF tag’s memory before inputting asset 
data into the RF tag. This process was a one-time action that was conducted by the TM staff and ATOS 
participants durmg pallet receipt. The warfighter stated that “The tag must be reusable, we must have options 
to rewrite the tag for different items.” Figure D-6 shows the response to the questionnaire statement about the 
on-command memory reset. 





Figure D-6. Questionnaire Results, On-Command Memory Reset: 27 of 29 respondents agreed the 
on-command memory reset was acceptable. 


MOP 1.3.5 Percentage of RF tags properly identified by their assigned ID number 

Dunng the Miesau and CAAA demonstrations, the warfighters assigned ID numbers to 20 RF tags. The 

were able to identify all 20 RF tags using the HHP. During the maritime demonstration, the warfighters assigned ID 
numbers to 60 RF tags. The warfighters were able to properly identify all 60 RF tags by their assigned tag ID numbers. 
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MOP 1.3.6 RF tag identifier humanly visible 


The assessment team observed that the warfighters were able to visually identify 20 selected RF tags by their ID 
numbers at each demonstration. No distance requirements was stipulated in the FRD. One warfighter stated that the 
“ID numbers were readable and easily located.” Figure D-7 shows responses to the questionnaires statement 
about identifying RF tags unique ID mumber. 





Figure D-7. Questionnaire Results, Identifying the RF Tags’ ID Number: All responses 
agreed that the unique ID number was identifiable on the RF tags. 


MOP 1.3.7 RF tag barcode readable 


The HHR was able to read the 200 RF tag barcodes at each demonstration site, but based on assessment team 
observations. the HHR had some problems reading the RF tag barcode if the warfighter didn’t place the HHR 
close enough to the RF tag to read the barcode. Several warfighters stated, “[The] HHR had some difficulty 


reading tags consistently.” Figure D-8 shows responses to the questionnaire statement about the readability of 
the RF tag barcode. 





Figure D-8. Questionnaire Results, RF Tag Barcode Readability: All responses agreed that 
the RF tag barcode readability was acceptable. 
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MOP 1.3.8 RF tag low battery power indication [backup battery] 


The TECEVAL data revealed that 20 RF tags switched over to their backup batteries and sent battery flags to the 
PP when their main cells fell below 3.2 volts. During the maritime demonstrations. the assessment team ob- 
served one RF tag transmitted a battery flag. Some of the warfighters indicated, “The RF tag needed a small 
light emitting diode or a test port to check the amount of electrical power left in the battery.” 

MOP 1.3.9 Operator assessment of the RF tag low battery power indicator 


The assessment team received positive comments about the RF tag low battery power indicator (Figure D-9). 
Some warfighters felt it would be beneficial to have a visual indicator on the RF tag itself that displays when the 
battery has switched over to the backup [battery] rather than having to check the PP for a flag indication. This 
would allow more time to change out the battery and not risk losing vital information. Other warfighters recom- 
mended that the RF tags should have a visual indicator on the outside of the RF tag casing. The questionnaire 
Tesponses indicated the RF tag low battery power indicator was acceptable for monitoring the battery power 
level on the RF tag. 


Rate the acceptability of RF tag low 
battery power indication 


g 
o 1 
2. 





Figure D-9. Questionnaire Results, RF Tag Low Battery Power Indicator: 25 of 27 respondents 
agreed that the RF tag low battery power indicator was acceptable. 


MOP 1.3.10 Detection range between RF tags and HHR 


The maximum detection range between RF tags and the HHR was approximately 80 feet based on assessment 
team observations. 


MOP 1.3.11 Detection range between RF tags and RCU 

The TECEVAL data indicated the maximum detection range for the RCU was approximately 100 feet. 
MOP 1.3.12 RF tag transmitter radiation parameters 

The RF tag transmitter power was at 240 milliwatts (mW), which met HERO requirements. 

MOP 1.3.13 RF tag transmitter frequencies 

The RF tag transmutter frequency was 433.92 Megahertz (MHz), which met HERO requirements. 
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MOP 1.3.14 Anti-collision communications between RF tag, HHR, and RCU 
This MOP assesses any communication conflicts between ATOS components. During the CAAA and maritime 
demonstrations, the assessment team observed interoperability problems between the HHR and RCU. The 
assessment team observed that the HHR did not work while the RCU was interrogating RF tags. One HHR 
would lock up if another HHR were in close proximity. 

MOP 1.3.15 Half-duplex communications between RF tag, HHR, and RCU 


The assessment team observed a few problems with data transfer between HHR and PP. During the maritime 
demonstration on the USS Harry S. Truman, the system was not successful in transfermng 180 transaction 
records from the HHR to the PP. In addition, ATOS was not successful in tansferring an inventory list that 
contained data from 200 RF tags from the PP to the HHR. 


MOP 1.3.16 Operator/SME assessment of their ability to change the RF tag 
user-selectable temperature limit from HHR 
The questionnaire responses indicated changing the RF tag user- 
selectable temperature was acceptable and easy using the HHR 
(Figure D-10). 








Figure D-10. Questionnaire Results, Change RF Tag Temperature Limit Settings: All respondents 
agreed that changing the RF tag temperature limits using the HHR was acceptable. 
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MOP 1.3.17 Operator/SME assessment of their ability to change RF tag 
user-selectable relative humidity limit from HHR 


Questionnaire responses indicated changing the RF tag user- 
selectable RH limit was acceptable and easy using the HHR 
(Figure D-11). 





Figure D-11. Questionnaire Results, Change RF Tag RH Limit Settings: All respondents agreed 
that changing the RF tags’ RH limits using the HHR was acceptable. 
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MOP 1.3.18 Operator/SME assessment of the reader capability to obtain RF 
tag maps ID number 


indicated obtaining the RF tag unique ID number using the HHR was effective (Figure 
D-12). Logistics and munitions SMEs indicated that the HER would sometimes have difficulty picking up the RF 
tag ID number from the HHR but would eventually read it. One logistics SME stated, “J think the capability is 
there but needs some improvement [in] this area.” 





Figure D-12. Questionnaire Results, Obtain RF Tag Unique ID Number: 63 of 69 of respondents 
agreed that using the HHR or RCU to obtain the RF tag: unique ID was effective 
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MOP 1.3.19 Operator/SME assessment of RF tag/reader communications 
Questionaire results indicated that the HHR and RCU were acceptable for transferring RF tag data despite 
technical problems encountered during the demonstrations. Al- 

though the results were positive (Figure D-13), the warfighters “Process slow at times. HHR had prob- 
complained that the read and write processing speed ofthe HHR ems in reading RF tags.” 


needs improvement. Warfighters 


Rate the acceptability of transferring RF tag data 
to the HHR and RCU 





Figure D-13. Questionnaire Results, Transfer RF Tag Data to the HHR and RCU: 61 of 64 
respondents agreed that the HHR or RCU was acceptable for transferring RF tag data 


MOE 1.4 Capability of ATOS HHR and RCU to communicate and 
transfer data to PP 


This MOE assesses the capability of the RCU and HHR to transfer stored RF tag data to the PP. During pallet 
Teceipt, the HHR was not able to transfer a large number of transaction records (1.e., 180 out of 200 transaction 
records) to the PP. as the HHR software would lock up after a couple minutes. During the CAAA demonstra- 
tion, the assessment team observed that when the warfighters docked the HHR., some transaction records did not 
get transferred to the PP and other transaction records were deleted from the HHR. To resolve the problem, the 
TM reduced the number of transaction records that were transmitted to the PP to only 50 transaction records at 
a time. 








MOP 1.4.1] Percentage of HHRs that were able to transfer data to PP 


Duning the Miesau demonstrations. all four HHRs transferred transaction records to the PP during pallet receipt. 
The assessment team compared the count of the handwaitten inventory list to the count of the PP inventory list. 
This comparison was accomplished at all demonstration sites. 


Duming the CAAA demonstration, all three HHRs transferred transaction records to the PP during pallet receipt. 
The assessment team observed technical problems with the data transfer of transaction records from the HHR 
to the PP. This was the first time the assessment team observed a problem with the RF tags having corrupted 
serialized data on the RF tags, which required the TM to purge the PP database. 


Dunng the demonstration onboard the USS Ponce. all three HHRs transferred transaction records to the PP 
during pallet receipt. A total of 60 out of 60 transaction records were successfully transferred to the PP. 
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Four HHRs were taken to the demonstration onboard the USS Harry S. Truman: however one of the four HHRs 
was not fully charged and therefore not available for assessment. Of the three remaining HHRs, all were able to 
transfer records to the PP during pallet receipt. During this demonstration, the assessment team leamed that the 
HHR was not able to transfer 180 transaction records to the PP. The TM developed a work around solution to 
transmit smaller packets (50 transaction records per packet) to the PP, and these transmissions were successful. 
One warfighter stated, “There were still some glitches in download process. Data tansfer/writing capabili- 
ties are sluggish. Would like to have faster processing of information.” Figure D-14 shows responses to 
the effectiveness of data transfer from the HHR to PP. 


Rate the effectiveness of data transfer from HHR to PP 
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Figure D-14. Questionnaire Results, Effectiveness of Data Transfer from HHR to PP: All 
respondents agreed that the data transfer from the HHR to the PP was effective 


Figure D-15 shows responses to the questiomnaire statement about ATOS supporting data transfer from 
HER to PP. 
Rate the acceptability of the ATOS system to support 
data transfer from the HHR to the PP 


25 
20 


g 
§ 
o 
& 
61 
a 
= 
5 
z 


o OW” 


Completely Largely Somewhat ir 
Unacceptable Unacceptable Unacceptable Acceptable Acceptable — Acceptable 


Figure D-15. Questionnaire Results, ATOS System Supports Data Transfer from the HHR to the 
PP: All respondents agreed the HHR to the PP transfer was acceptable. 
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MOP 1.4.2 Percentage of RCUs that were able to communicate and transfer 
data to PP. 


Both RCUs used during the Miesau demonstration were able to communicate data to the PP. The RCU im the 
earth-covered magazine transferred interrogated RF tag data to the PP via WLAN. The second RCU trans- 
ferred data via a serial port. This RCU had a data transfer problem when communicating with the HAR, which 
was resolved by a software change. The assessment team observed the transfer of the interrogated RF tag data 
to the PP by using the HHR and verified that the PP inventory matched the list of RF tags placed inside the test 
site. The assessment team also verified the pallet count changes were updated on the PP. 

Two RCUs were used in the CAAA demonstration. One RCU was installed in an earth-covered magazine and 
the other was installed in a storage warehouse. Both RCUs were connected to a PP via WLAN and both were 
successful in transferring interrogated RF data to the PP. The assessment team verified that the PP inventory 
matched the list of RF tags placed inside the test sites and that the pallet count changes were updated on the PP. 
One RCU was used in the maritime demonstration on the USS Harry S. Truman, and was successful in transfer- 
Ting interrogated RF tag data to the PP viaa LAN. The assessment team verified that the PP inventory matched 
the list of RF tags placed inside the test site and the pallet count changes were updated on the PP. Most of the 
warfighters indicated, “Slow processing speed does not always pick up the RF tag information. Overail, it 
seems, to work ok.” Figure D-16 shows responses to the questionnaire statement about data transfer for 
several RF tags to the RCU and PP. 


Rate the acceptability of the ATOS system to support 
data transfer from several RF Tags to the RCU and PP 
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Figure D-16. Questionnaire Results, ATOS System Supports Data Transfer from Several RF Tags 
to the RCU and PP: All respondents agreed the data transfer of the RF tags to the RCU and PP 
was acceptable. 
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MOP 1.4.3 SME assessment of HHR-PP connectivity 
Questionnaire responses indicated that the HHR-to-PP connectivity was usable during the demonstrations (Fig- 
ure D-17); however, the assessment team observed several data transfer problems during pallet receipt. One 
warfighter stated. “[It] seems when docking the HHR there was some problems with data transfer.” Al- 
though the HHR-to-PP data transfer process was usable, the warfighters stated it needed improvement. 


Rate the usability of the HHR-PP connectivity 
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Figure D-17. Questionnaire Results, HHR-PP Connectivity: All respondents agreed that the HHR- 
PP connectivity was usable. 


MOP 1.4.4 WLAN transmitter radiation parameters 

The WLAN power was 100mW effective isotropic radiated power, which meets the HERO requirement for a 
COTS product. 

MOP 1.4.5 WLAN transmitter frequencies 


The WLAN transmitter frequency was 2.5 Gigahertz. 

MOP 1.4.6 Detection range for WLAN between RCU and PP 

At the Miesau demonstration, a line-of-sight connection was established with the RCU inside the earth-covered 
magazine and the PP. which was located an eighth of a mile away. The assessment team did not observe any 
issues with RCU and PP connectivity using the WLAN. 

At the CAAA demonstration, two WLANs were installed at the test sites. The first WLAN required a repeater 
on top of a 75-foot water tower to establish a 1.5 mule connection to the PP. The second WLAN was a line-of- 
sight connection that was less than 0.5 miles away from the PP. At both test sites, the WLAN was able to 


provide reliable comnectivity from the RCU to the PP throughout the demonstration. 
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MOP 1.4.7 Operator assessment of HHR, RCU, and PP communication and 
data transfer interface 

transfer interface between the HHR. RCU, and PP was ac- 
ceptable for the demonstrations (Figure D-18); however, most — 
of the warfighters believed the data tansfer interface between 
the HHR, RCU, and PP needs improvement. 








Figure D-18. Questionnaire Results, HHR, RCU, and PP Communication and Data Transfer: All 
respondents agreed that the HHR, RCU, and PP connectivity waz acceptable. 


MOE 1.5 Capability of ATOS PP to interoperate with Service 
Ammunition AIS 

This MOE was not assessed because the Service Ammumition AIS was not available during the demonstrations. 
MOE 1.6 Capability of ATOS PP to interoperate with EDB 

This MOE was not assessed because the EDB was not available during the demonstrations. 

MOE 1.7 Characterization of network bandwidth connectivity to 
ATOS PP 


This MOE assesses the data connectivity of the PP from the HHR and RCU. The HHR docking station was 
connected to the PP by an Ethemet connection. A LAN or a WLAN were used to connect the RCU to the PP. 
During the Miesau, CAAA, and mantime demonstrations, the assessment team did not observe any network 
bandwidth problems that would impact data transfer from the RCU to the PP or from the HHR and PP. 








D-20 Advanced Technology Ordnance Surveillance Final Report 
For Official Use Only 


140 


MOP 1.7.1 Bandwidth between HHR and PP 


The HHR docking station was comnected to the PP by an Ethemet connection. The bandwidth was 250 Mega- 
bytes (MB)/second. When the HHR was placed into the docking station, the data were automatically down- 
loaded to the PP. As stated earlier, the TM needed to fix the software on the HHR and PP to resolve data 
transfer problems encountered during the demonstration. The assessment team did not observe any data transfer 
problems involving network bandwidth. 


MOP 1.7.2 Bandwidth between RCU and PP 


The assessment team observed no problems with network bandwidth between the RCU and PP. The TECEVAL 
data revealed the WLAN bandwidth was 10 MB/second. which is the commercial standard for a WLAN: and 
250 MB/second via Ethemet comnection. 


MOP 1.7.3 Operator assessment of bandwidth between HHR and PP 


The operators had insufficient knowledge to assess bandwidth. The assessment team did not collect any band- 
width data during the demonstration. 


MOP 1.7.4 Operator assessment of bandwidth between RCU and PP 


The operators had insufficient knowledge to assess bandwidth. The assessment team did not collect any band- 
width data during the demonstration. 


MOE 1.8 Capability of ATOS system to protect data 


The assessment team was unable to assess network security since a security SME was not available during the 
demonstrations. 


MOE 1.9 Capability of ATOS system to provide sufficient data 
storage 


This MOE assesses the data storage capability of the HHR, RCU, and PP. The assessment team did not observe 
any data storage issues with ATOS during the demonstrations. The HERO requirements for the demonstrations 
restricted the number of RF tags that could be used to support the assessment: as a result, only 200 RF tags could 
be used for each test facility per demonstration. 


MOP 1.9.1 HHR storage memory capacity 


The HHR storage memory capacity was 64 MB of read-only memory and 64 MB random-access memory. The 
assessment team did not observe any data storage issues during the demonstrations. 


MOP 1.9.2 RCU storage memory capacity 

The RCU storage memory capacity was 256 MB. The assessment team did not observe any data storage issues 
dunng the demonstrations. 

MOP 1.9.3 PP storage memory capacity 


The PP storage memory capacity was based on the laptop computer hard drive storage capacity. which was over 
40 Gigabytes. PP storage capacity was never an issue during the demonstrations. 


————————————————————————————————————————————————— 
Advanced Technology Ordnance Surveillance Final Report D-21 


For Official Use Only 


141 


MOP 1.9.4 SME assessment of RF tag data storage 


Questionnaire results indicated the warfighters believed the RF tag data storage was usable and adequate to 
support munitions management (Figure D-19). Most of the warfighters agreed that having RF tags with large 
memory would be useful for changing and storing additional information about the tagged mumitions pallets. 


Rate the acceptability of the RF Tag data storage 
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Figure D-19. Questionnaire Results, RF Tag Data Storage: All respondents agreed that the RF 
tag data storage was acceptable. 


MOP 1.9.5 SME assessment of data storage for HHR and RCU 


Questionnaire results indicated the SMEs believed the HHR and RCU data storage was usable and adequate to 
support munitions management (Figure D-20). ATOS data storage capacity was never exceeded during any of 
the demonstrations. 


Rate the acceptability of the HHR and RCU data storage 
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Figure D-20. Questionnaire Results, HHR and RCU Data Storage: All the SMEs agreed that the 
HHR and RCU data storage was acceptable. 
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MOP 1.9.6 SME assessment of PP data storage 


The questionnaire results indicated that the PP data storage was usable and adequate to support munitions 
management at the Miesau and CAAA demonstrations (Figure D-21). Only the accountable record managers 
responded to this questionnaire statement. 


Rate the acceptability of the PP data storage 
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Figure D-21. Questionnaire Results, PP Data Storage: All SMEs agreed that the PP data storage 
was adequate. 


MOE 1.10 Characterization of ATOS RCU antenna coverage 


This MOE assesses antenna coverage of the RCU inside an earth-covered magazine, storage warehouse, or 
shipboard magazine. The RCU and RFE were effective in providing adequate antenna coverage to detect 200 
RF tags during the demonstrations. 

Dunng the first mterrogation cycle at the Miesau demonstration. the RCU detected 199 out of 200 RF tags 
(99.5%) ina 60x 40 x 15 foot earth-covered magazine with three RFEs; 169 out of 174 tags (97.1%) ina 70x 200 
foot ASP with eight RFEs. During the second and third interrogation cycles, the RCU detected 100% of the RF 
tags. The warfighters expressed some concems about the application of ATOS in an ASP. One Miesau warfighter 
stated, “For an enclosed location the system seems to work well. The problem I see is in what was de- 
scribed as the ASP application. In an open area, the requirement could be extensive, depending on the 
size of the area to be covered.” Another concem is that a ‘ring of antennas’ around a storage area may not be 
possible due to terrain restrictions. 

Dunng the CAAA demonstration, the RCU detected 197 out of 200 tags (98.5%) in both the 100 x 50 x 10 foot 
earth-covered magazine using 6 RFEs and the 200 x 50 foot storage warehouse using 12 RFEs. Dumng the 
maritime demonstration, the RCU detected 60 out of 60 RF tags (100%) in a 10 x 22 x 8 foot magazine on the 
USS Ponce and 200 out of 200 RF tags (100%) in a 45 x 80x 10 foot magazine using 3 RFEs on the USS Harry 
S. Truman. 


MOP 1.10.1 Antenna coverage between RCU and RF tag 


Dung all demonstrations, the RCU detected approximately 98% of the RF tags on the first interrogation cycle. 
and 100% after two or three interrogation cycles. 
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MOP 1.10.2 Operator/SME assessment of antenna coverage between RCU 
and RF tag 


Questiomnaire results indicated that the RCU and RFEs were acceptable and provided adequate antenna cover- 
age inside the magazine (Figure D-22). 





Figure D-22. Questionnaire Results, Antenna Coverage between the RCU and RF Tag: All 
respondents agreed that the antenna coverage was acceptable during the demonstrations. 


MOE 1.11 Capability of ATOS RCU to detect RF tags at initial 
startu 
This MOE assesses the performance of the imitial detection of the RF tags by the RCU. During the Miesau and 


CAAA demonstrations, the RCU was able to detect more than 97% of 200 RF tags, and during the maritime 
demonstration detected 100% of 200 RF tags. 


MOP 1.11.1 Initial RF tag detection using RCU within 145 minutes 


The ATOS system was able to detect over 97% of the RF tags within 14 munutes of the first interrogation cycle 
during the operational demonstrations. At the Miesau demonstration, the first RCU detected 199 out of 200 RF 
tags (99.5%) during the first interrogation and the ASP RCU detected 194 out of 200 RF tags (97.1%). At the 
CAAA demonstration, the RCU detected 197 out of 200 RF tags (98.5%) during the first interrogation inside the 
earth-covered magazine and storage warehouse. At the mantime demonstration on the USS Harry S. Truman. 
the RCU detected 200 out of 200 RF tags during the first interrogation inside the ship magazine. 

MOP 1.11.2 Initial RF tag detection using RCU within 215 minutes 


Not assessed. The assessment team was not able to verify missing or new RF tags were detected within 215 
minutes based on how the ATOS system interrogates RF tags using the short and long read cycles. 
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MOE 1.12 Usability of ATOS HHR input devices 

This MOE assesses the operators’ ability to use the keypad and stylus to input data on the HHR. Dumng all 
demonstrations, the operators were able to use the HHR keypad and stylus to input asset data. Most of the 
operators preferred using the liquid crystal display (LCD) keypad rather than the HHR keypad to input asset 
data. Even when wearing bulky gloves. the operators were able to edit asset information and perform the 
munitions management tasks using the HHR keypad. 

MOP 1.12.1 HHR keypad input 

The questionnaire responses indicate that the HHR keypad was usable to all the respondents (Figure D-23). 
Dung pallet receipt, pallet transfer, and inventory maintenance, warfighters preferred to use the LCD keypad 
and stylus to input asset information data on the HHR. Questionnaire responses indicated the HHR keypad was 
usable for data entry (Figure D-24). 


Rate the usability of the HHR keypad 
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Figure D-23. Questionnaire Results, Usability of HHR Keypad: All respondents agreed that 
the HHR keypad was usable 


Rate the usability of the HHR data entry 
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Figure D-24. Questionnaire Results, Usability of HHR Data Entry: All respondents agreed 
that the HHR was usable for data entry 
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MOP 1.12.2 HHR stylus input 
The warfighters were able to use the stylus to imput asset information (Figure D-25) and they found it especially 
useful when wearing bulky gloves. 





Figure D-25. Questionnaire Results, Usability of HHR Stylus: 32 of 33 respondents agreed that 
the HHR stylus was usable. 
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MOE 1.13 Capability of ATOS HHR to create RF tags and 
barcodes 


This MOE assesses the capability to use the HHR to create new RF tags and barcodes. During the demonstra- 
tion, the HHR was not used to create barcodes. Instead, the barcode template was created by the TM using a 
barcode machine. Warfighters were able to use the HHR to input asset information on new RF tags using 
barcode templates. Most of the warfighters believed the process was easy and straightforward, although some 
complained that it took two to five minutes to transfer asset data 

from the HHR to the RF tag. 
MOP 1.13.1 HHR creation of new RF tags 
At each demonstration site. the warfighters were given 20 RF — 
tags to capete, The wariighiiers weve able to quickly. populate the ikF tagt with entet dita: Quetlionnaive se- 
sponses indicated that creating a new RF tag using an HHR was acceptable (Figure D-26). The two unaccept- 
able comments did not provide any system improvement information. 











Figure D-26. Questionnaire Results, HHR Creation of New RF Tags: 31 of 33 respondents 
agreed that using the HHR to create new RF tags was acceptable. 
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MOP 1.13.2 HHR creation of replacement RF tags 


At each demonstration site, warfighters replaced five RF tags using the HHR The warfighter was able to 
transfer the asset information from the old RF tag to the replacement RF tag using the HHR. Questionnaire 
responses indicated that creating a replacement RF tag using an HHR was acceptable (Figure D-27). 


Rate the ability of the HHR to create 
replacement RF Tags 





Figure D-27. HHR Creation of Replacement RF Tags Questionnaire Results: 31 of 33 
respondents agreed that using the HHR to create replacement RF tags was acceptable. 


MOP 1.13.3 HHR creation of barcodes 
Not assessed. The capability was not used during the ACTD. 


MOE 2.1 Reliability of ATOS RF tag attachment 

This MOE assesses the RF tag attachment method used dunng the demonstrations. Dunng the Miesau and 
CAAA demonstrations, the RF tags were not attached to the munitions pallets but were placed on the side or on 
top of the munitions pallets. A limited assessment was conducted by placing the RF tag holders on one wooden 
pallet and one metal pallet. Two methods were used to attach the RF tags on the wood and metal munitions 
pallets. The first method used wood screws to attach the tag holder, while the second method used a special 
double-sided adhesive tape. The adhesive tape was very effective on metal pallets. Although the adhesive tape 
was used on a wooden pallet. the warfighters recommended that wood screws be used to secure the tag holder. 
TECEVAL data revealed that the RF tag could be easily separated from the tag holder if a mass of 790 grams 
was placed on top of the RF tag. A similar shear force from either side or the bottom also would cause the RF tag 
to separate from the holder. Although the RF tags remained in place during the demonstrations, one warfighter 
stated, “The RF tags needed a magnetic interface to put up a flag when RF tag is removed from tag holder. 
This would eliminate questionable moves.” 

MOP 2.1.1 Reliability of attachment method 


Insufficient data were collected to assess the reliability of the attachment method used to attach the tag holder to 
the mumitions pallet dunng transport. 


MOP 2.1.2 Reliability of RF tag attachment 
Insufficient data were collected to assess the reliability of the RF tag remaining attached to the tag holder dunng 
transport. 
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MOP 2.1.3 Operator assessment of RF tag attachment and attachment 
method 


Questionnaire responses indicated tag attachment and attachment methods were acceptable. but needed some 
improvement (Figure D-28). One USS Harry S. Truman warfighter felt the RF tag holder should be made out 
of the pallet material to improve durability. Three CAAA warfighters did not like the idea of using adhesive tape 
to attach the RF tag holders to a wooden box or munitions pallets. Warfighters strongly recommended using wood 
screws to secure the RF tag holders to the wooden box or pallets to prevent the tag holder from being pulled off 
during transport. 


Rate the acceptability of the RF Tag attachments 
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Figure D-28. Questionnaire Results, RF Tag Attachment and Attachment Method: 22 of 28 
respondents agreed that the RF tag attachment and attachment method were acceptable. 


MOE 2.2 Usability of ATOS HHR and RCU 


EE 
This MOE assesses the usability of the HHR display, HHR data entry, HHR docking station capability, and the 
RCU-PP bridging using the Ethemet connection. The warfighters felt the HHR and RCU were easy to use and 
operate, and a time-saver for managing the movement and shipment of munitions pallets. One Miesau warfighter 
stated that. “Once the HHR is fixed, the HHR should be highly usable and user friendly.” An operator 
stated, “Display is user fiiendly.”’ The warfighters were able to download a partial inventory list of 20 to 60 RF 
tags from the PP to the HHR. The HHR was taken to the magazine or storage location to conduct the inventory 
and identified new or missing tagged pallets within 15 mmutes. During the demonstrations, the assessment team 
observed one data transfer problem with the HHR-RCU bridging capability and two data transfer problems with 
the HHR-PP bridging capability volving transmission of 200+ RF tag data records. The HHR-RCU bridging 
was resolved by a software fix and demonstrated at CAAA. The HHR-PP data transfer problem occurred 
during the maritime demonstration. This problem raised some concems about accuracy of the inventory list 
during the data transfer process. The warfighters also expressed some concerns about the HHR processing time 
for reading and writing data to the RF tags. All warfighters felt it took a long time to process an RF tag. 


OO  ————. 
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MOP 2.2.1 HHR display usability 
The warfighters felt the HHR display was user fnendly and easy to use (Figure D-29). 





Figure D-29. Questionnaire Results, Usability of the HHR Display: All respondents agreed that 
the HHR display was usable. 
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MOP 2.2.2 HHR input usability 
The warfighters stated the HHR data entry was easy (Figure D-30). Reference MOP 1.12.1 for the results. 





Figure D-30. Questionnaire Results, Usability of the HHR Data Input: All the respondents 
agreed about the usability of the HHR to input data. 
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MOP 2.2.3 HHR-RCU bridge usability 
(Ethernet connection) atl 
After the TM fixed the software bugs, the HHRandRCUbndge < 

worked well in transferring data from the RCU to the HHR 
indicated the HHR-RCU bridge was usable to support mumi- 
tions management (Figure 31). The unusable comments came from the Miseau demonstration. These com- 








Figure D-31. Questionnaire Results, Usability of the HHR-RCU Bridge: 22 of 24 respondents 
agreed that the HHR-RCU bridge was usable. 
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MOP 2.2.4 HHR-PP bridge usability 

The assessment team observed data transfer problems with the HHR-PP bridge. During the maritime demon- 
stration, the HHR transferred all the transaction records to the PP during pallet receipt, but the PP did not transfer 
the inventory list of 200 RF tags to the HHR. During the Miesau and CAAA demonstrations. the HHR and PP 
were successful in downloading data from 20 to 60 RF tags. Questionnaire responses indicated the HHR-PP 
bridge was usable (Figure D-32). 





Figure D-32. Questionnaire Results, Usability of the HHR-PP Bridge: 
that the HHR-PP bridge was usable. 


All respondents agreed 
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MOP 2.2.5 HHR-PP docking station usability 


Warfighters were able to dock the HHR and transfer the RF tag data to the PP when ATOS did not encounter 
data transfer problems (Figure D-33). 





Figure D-33. Questionnaire Results, Usability of the HHR-PP Docking Station: All respondents 
agreed that the HHR-PP docking station was usable. 


a. 
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MOE 2.3 Usability of ATOS PP 


eae ese eee ee ee eT 
This MOE assesses the PP display and ease of data entry. The warfighters indicated the PP was easy to leam 
and operate and had user-fnendly menus. 


MOP 2.3.1 PP display usability 


The warfighters indicated the PP display was easy to use to obtain inventory and transaction information (Figure 
D-34). 





Figure D-34. Questionnaire Results, Usability of the PP Display: All respondents agreed that 
the PP display was usable. 
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MOP 2.3.2 PP input usability 
The warfighters stated that data entry input using the PP was easy and had user-friendly memus (Figure D-35). 








Figure D-35. Questionnaire Results, Usability of the PP Data Entry: All respondents agreed that 
using the PP was usable to input data 


MOE 2.4 Ease of configuration for ATOS components 

This MOE assesses the operators” ability to easily configure the RF tags, HHR, RCU, and PP during the demon- 
stration. ATOS components were easy to configure and could be adjusted quickly to meet the warfighter’s 
needs. One warfighter stated, “[He] had no problem configuring RF tags.” Another warfighter stated, “The 
HAR and the PP were easy to learn and have user-friendly software interfaces." Although most comments 
were positive about ATOS components, the assessment team received several comments about potential im- 
provements. A Miesau warfighter stated the RF rags need to be reusable.” Due to the cost of making the RF 
tag, the warfighter wanted recyclable RF tags. Another CAAA warfighter felt the RF tags should have the 
capability to read a 1348 barcode (inventory sheet). The warfighter can print a barcode and wnite directly to the 
RF tag using an HHR. 
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MOP 2.4.1 Capability to configure RF tag 


The warfighters were able to use the HHR to configure the RF 
tags by entering asset information and adjusting user-selectable 
environmental settings (Figure D-36). The unacceptable com- 
ment did not provide any system improvement information. The unacceptable comment did not provide any 
system Improvement regarding the configuration of the RF tags. 








Figure D-36. Questionnaire Results, Configuring RF Tag: 02 of 63 respondents agreed that 
configuring the RF tag was acceptable. 
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MOP 2.4.2 Capability to configure HHR “Very easy to use and configure” 


The warfighters were able to quickly configure the HHR to link Warfight 
with the PP (Figure D-37). The unacceptable comment did not 


provide any system improvement information. 


Rate the acceptability of configuring the HHR 


fo accommodate the grester number ct responses, the graph scale was increased. 
Figure D-37. Questionnaire Results, Configuring HHR: 01 of 02 respondents agreed that 
configuring the HHR was acceptable 





MOP 2.4.3 Capability to configure RCU 

The assessment team did not observe the ATOS operators making any configuration changes to the RCU. 
MOP 2.4.4 Capability to configure PP 

The assessment team did not observe the ATOS operators making any configuration changes to the PP. 


MOE 2.5 Adequacy of ATOS training 


This MOE assesses the training received at the demonstration sites. The warfighters agreed traming was 
adequate after using ATOS. The traiming was effective in showing the warfighters how to operate the system. 
One warfighter stated, “maining was excellent but could be a bit more structured. Try more visual aids (i.e. 
attach a tag to a pallet‘box and go through a pallet mansfer and pallet issue.)."" All warfighters wanted an 
ATOS user’s guide and training manual to explain the system operation. 
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MOP 2.5.1 Training for RF tag 
Training was adequate to show the ATOS participants how to use the RF tag (Figure D-38). 


Rate the effectiveness of RF Tag training 


: 
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Figure D-38. Questionnaire Results, RF Tag Training: All respondents agreed that the RF 
Training was effective. 
MOP 2.5.2 Training for HHR 


Training was adequate to show the ATOS participants how to use the HHR (Figure D-39). The ineffective 
comment was focused on the amount of material presented during the taining. 


Rate the effectiveness of HHR training 
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Figure D-39. Questionnaire Results, HHR Training: 55 of 56 respondents agreed the HHR 
training was effective. 
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MOP 2.5.3 Training for RCU 
Training was adequate to show the ATOS participants how to use the RCU (Figure D-40). The ineffective 
comment was focused on the amount of matenal presented dunng the training. 


Rate the effectiveness of RCU training 


& 


Number of Responses 


2 accommodate the greater number of responses, the graph scale was Increased. 





Figure D-40. Questionnaire Results, RCU Training: 50 of 51 respondents agreed the RCU 
training was effective. 


MOP 2.5.4 Training for PP 
Training was adequate to show the ATOS participants how to use the PP (Figure D-41). 


Rate the effectiveness of PP training 
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Ineffective Ineffective 
0 accommocate the greater number of responses, the grapn scale was increased. 





Figure D-41. Questionnaire Results, PP Training: All respondents agreed that the PP training 
was effective. 
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MOP 2.5.5 Effectiveness of ATOS training material 

The warfighters felt the traming was effective in providing the necessary information to operate the ATOS 
system during the demonstrations (Figure D-42). The meffective comment did not provide any information on 
improving the taming. 


Rate the effectiveness of the ATOS training material 





Figure D-42. Questionnaire Results, ATOS Training Material: 55 of 56 respondents agreed the 
ATOS training material was affective 


MOE 2.6 Adequacy of ATOS component ruggedness 


This MOE assesses the ruggedness of the RF tags, RCU. and HHR im an operational environment. The assess- 
ment of the ATOS component ruggedness was limited: however, TECEVAL data revealed the RF tags met the 
Tuggedness requirements stipulated in the FRD. All RF tags performed nomunally except for two inoperable RF 
tags (cause unknown). The assessment team did not observe any HHR failures due to handling by the warfighters: 
however, the assessment team observed the HHRs lock up on occasion and needed to be rebooted. The 
assessment team also observed two RCU failures. One failure was due to a faulty flash memory card and the 
second failure was due to a bad power supply. With the exception of the data transfer problems with the HHR 
and PP. ATOS performed nomunally during the demonstration. 

MOP 2.6.1 RF tag ruggedness 

TECEVAL data revealed the RF tags met the ruggedness requirements stipulated in the FRD. The RF tags can 
operate between -20°F and +155°F and between 0% and 95% relative humidity (non-condensing). The RF tags 
functioned after a 15 kilovolt electrostatic discharge test and 4-foot drop onto concrete. The RF tags operated for 


10 days at the Miesau and CAAA demonstrations where they were placed outdoors. RF tags were functional 
when they were shipped from CAAA to Norfolk Naval Station via commercial ground transportation. 


MOP 2.6.2 HHR ruggedness 

TECEVAL data revealed the HHR met the ruggedness requirements stipulated in the FRD_ The assessment 
team observed no HHR failures during the demonstrations. 

MOP 2.6.3 RCU ruggedness 

TECEVAL data revealed that the RCU met the mggedness requirements stipulated im the FRD. The assessment 
team observed two RCU failures during the demonstrations. The cause for the failures has not been determined. 
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MOE 2.7 Supportability of ATOS power requirements 

This MOE assesses the power requirements for the RF tags, HHR, RCU, and PP. The assessment team observed no 
miajor issues with providing electrical power to support ATOS components during the demonstrations. Some CAAA 
users felt that providing electrical power to remote storage magazines or facilities could be a challenge. However. 24 
VDC battenes or gas-powered electric generators could be used to support power needs at these remote sites. 
MOP 2.7.1 Voltage for RF tag operation 

No RF tags were replaced due to low battery power indication during the demonstrations. 

MOP 2.7.2 Voitage for HHR operation 

The HHR used two rechargeable battenes and docking stations to ensure power was maintained. The assessment 
team did not observe any power issues with the HHR. 

MOP 2.7.3 Voltage for RCU operation 

As long as the RCU was able to obtain 24 VDC, the RCU was able to operate without any problems. 

MOP 2.7.4 Voltage for PP operation 

The PP was hosted on a commercial laptop computer. As long as 120/220 VAC power was available, the PP 
continued to work. 

MOP 2.7.5 Detection of type of electrical input power to RCU 
Cee Cert ee nee ee 
Multipurpose Wheeled Vehicle battery, or 120 VAC line power using a 24 VDC step down 

MOP 2.7.6 Operator/SME assessment of RF tag power requirements 

The Miesau mmitions SME indicated the RF tag batteries should — 
bereplaceable instead of throwing away the RF tag. The warfighter 
wanted to see a visual indicator on the outside of the RF tag casing 
to display battery strength. Questionnaire responses indicated the 
RF tag power requirement was acceptable dunng the demonstra- 
tion (Figure D-43). The unacceptable comments focused on the 
designing of the RF tags so that the operator can replace the batteries. 









igure D-43. Questionnaire Results, RF Tag Power Requirements: 51 of 59 respondents 
agreed that the RF tag power requirements were acceptable. 
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MOP 2.7.7 Operator/SME assessment of power requirements for HHR 
and RCU 


The CAAA warfighter indicated power availability at remote storage locations would be a major concem. Many 
of these magazines will not have electrical power to support the HHR and RCU power requirements. Question- 
naire responses indicated the HHR and RCU power requirements were acceptable during the demonstrations 
(Figure D-44). The unacceptable comment did not provide any information to improve the power requirement. 





Figure D-44 Questionnaire Results, HHR and RCU Power Requirements: 22 of 23 respondents 
agreed the HHR and RCU power requirements were acceptable. 
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MOP 2.7.8 Operator/SME assessment of PP power requirements 


The warfighters indicated the PP power requirements were straightforward and easy to accomplish as long as 
the PP was hosted in a facility that had 120/220 VAC electrical power (Figure D-45). 





Figure D-45. Questionnaire Results, PP Power Requirements: All respondents agreed that the 
PP power requirements were acceptable. 
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MOE 2.8 Supportability of ATOS installation 

This MOE assesses the installation of ATOS components. The installation at all the demonstration sites was 
quick and easy as long as there was a place to hang the RCU and RFEs and there was a 120/240 VAC power 
source for the RCU and RFEs. 


MOP 2.8.1 Operator/SME assessment of RCU installation 


The warfighters indicated the location and accessibility of the RCU were acceptable for the demonstrations as 
long as there was a place to hang the RCU and RFEs (Figure D-46)_ Tie wraps were used to install the RFEs 
and the RCUs were mounted on portable stands or a table. The unacceptable comment did not provide any 
information that would mprove the RCU installation. 








Figure D-46. Questionnaire Results, RCU Installation: 36 of 37 respondents agreed that the 
RCU installation was acceptable. 
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MOP 2.8.2 
The warfighter indicated the PP installation was straightforward since the software is hosted on a laptop. The 
questionnaire responses indicated the PP installation was acceptable during the demonstration (Figure D-47). 





Figure D-47_ Questionnaire Results, PP Installation: All respondents agreed the PP installation 
was acceptable. 
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As requested by USEUCOM, this appendix correlates FRD requirements. assessment results, and MOPs. 


Requirements 
Data Tag 


1.1 Attachment. The RF tag will be inserted 
into a holder, which wil be securely attached 
fo he pallet a container. 





1.1.1 Tag hdder must securdy attach to 


easy F 
remowrepacerert and can sue pa 


Ree The RF tag ‘will awe 
sufficent nawolafle memory to store ail tag 
dats items 

vanouaie two sensors, temperature and 
relative humidity, with ports to suppert two 
additional sensors far fulure expansion. Up to 
20 RF tags will contain a shock sensor to be 
used in parallel testing and evaluafon at 
NAVSEA, indian Head, MD for possible fue 
RF tag appiicafon. 


14 Shock Sensor. The shock sersar wil 


individually 
shall be two shock measures on each of the 
%, >, and Z-axes. in the eventof the tag 
expedencng a shock in excess of a sensor 
fresno, a data fag resident on the RF tag 
will be set to indicate the event. 

14.1 The failure rate for shock fumshold 


excessicn flags being incormctly set, or not 
sel, shall be no gmater than 5%. 
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Results 


During the dermors rafors , both RF tags and tag holders 
remained attached to the munifons pallets. The wadighters 
indicated the RF tag atlachnent and altachrnent methad used 
qm the metal containers woked wel. 
All 32 RF tags performed as designed. One RF tag hakler carne 
dif a munifons pallet during the marifime demonstafon. 
During the Miseau and Crane demonstafors, bof RF tags and 
tag hdders remained attached to the metal munifiors pallets. 
The indicated the RF tag attachment and attachnent 
method used on the meta containers worked well. Wariightes 
stongy recanmended using wooden screws to secure the RF 
ee ees ee ees 

d fd 


ik led AE bags omelet nace 
200 RF tags were used constanfy to support all the 
demonstrafons. Most of hese RF tags were inifialized in March 
2004 (approximately 4700 hours operating time). The 
a@seesenent lean did nol observe any Gilures in he temperature 
sersar and relative humidity sensors. 

The RH sensor was accurate within 2 10%. 

The RF tag can recard and store far 45 days (1080 dats points) 
of environmental data. 

The RF tag collected and recorded tenperature and relafive 
hurnidity data every 60 minutes. Confirm by reviewing he 
histograrn date. 

During the demarstrafions, the TM randomly selected the high 
and low user-selectable temperature and RH limits. Many of 
these settings were not qerafonally representative. Alfough 
fhe PP was atie to display tipped flags, fhe assesernent team 
did not know how many RF tags were used to demonstrate ths 
capability. 

During he demors rafors, both RF tags and tag holders 
rermained aitached to the munifons pallets. The warighters 
indicated the RF tag altachrnent and altachrnent method used 
m the metal containers worked wal. However, warfighters were 
concemed about measuring fhe temperature and RH inside the 
individual maunifions cantainers. Recommend RF tags be placed 
inside these containers rather fan on fhe muniicns pallets. 
When availatie, enwironmenta sersars shoud be embedded in 
individual munifons. 
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1.25 
1.28 
1.29 
1.2.10 
1.21 
1.2.12 
1.2.13 
1.2.14 





Requirements 


1.6.1_RH sensors shall be accurate 02 5 LRH sensor was accurate witin# 10% 00 


162 Tea ay aus dio re "The aaseaament team observed no tailures In fle RH senaoe apae ol 
i. —— 


temperature limits set. All 20 tags had ter environment flags 
vipped at he appmpdate lint setings. 
‘The assesemnent eam observed no failure in he tenperture 


accommodate for high and low humidity lierits. 
A bil fag or indicator assocated with 





Crane demonstmafon, RCU detected 197 aut af 200 tags 
(98.5%) in bath te 100 x 50’ x10 eart)-covered magazine 
using 6 RFEs and 200’ x 50’ slorage warehouse using 12 RFEs. 
Althe Maime demons ration, HHR detected 0 out of 60 RF 
tags (100%) in a 10’ x 22°x 8’ magazine an the USS Ponce 
(RCU was nat installed) and 200 aut of 200 RF tags in a 45 x 80° 
Oe ee ie ie es ae Se el 


salient "Til tag "oul ip all hn camneedy Gaz saci a rhe iat lan EM ace 

a receiver counter on board fie tag and willbe | activity. The RCU dala logge documented the start and stop 

set for hourly meadings. The “wake up" interval | Smes ofthe RCU. The msulls indicated hat fe RCU 

will be access ble for change via an appmpdate | autanafcally imerrogated the RF tags every60 minutes. When 

operator interface. the RCU detects a change in the RF tag asset information, the 
RCU sends a fag to he PP during he hourly interrogation. 
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Requirements Results Wey 


The antenna coverage was exceptional at the damonstatons. 1.10.1 
At the Miesau and CAAAdemonstators, the RCU was able to 
between RF tag and reader) in amagazine. | detect at least 98.5% of 200 RF tags during the frst intarmgaton 
cyde at both demorstators and after the third intarragafon 
cyde 100% of the RF tags ware deteciad. 


1.9.3 The unobstructed read/write range 13.10 
batwean the RF tag and a HHR shail be at 
tanat 20 feet. The faikre rate shall not exceed 


and RCU. The recaiver will be available every | The log&fcs and runifons SMEs indicated that he HHR woud | 1.2.4 
5 seconds sometmes have trouble picking up fhe RF tag ID number. Law 
The munitors SMEs felt changing the tag user-selectable 13.16 
temperatum was easy to lean and use with fhe HHR. 1317 
The munitors SMEs fait changing the RF tag user-selectatia 1218 
RH lirnit was easy to laarn and we with the HHR. 1.3.19 


20 RF tags switched over fo their backup batieries and sant 138 
batiary fags to the RCU when the main cells fall below 32 volts 2.7.1 
during TECEVAL During the demors tations, one RF tag 


271 


of displaying fhe RF tag ID. bean ree 136 
download the HHR fansacton records fo the PP to display he 137 
assigned RF tag ID. During he Miesau and CAAA 
Levees eaerblgh-aarvicbresaeheetclpreeien die -redarhy 
RF tags using the HHR. During the mastirne denonsimion, he 
warighters ware able to praperty idertify 60 cut of 60 RF tags 
(100%) by their assigned tag ID nurnbers. 
The warighters ware able to visually identify the 20 selaciad RF 
tags by their 1D numbers at each demonsimfon. One warightar 


seit il Al ie adi acl 1.13.2 
“burn,” a new tag using data stored an a warighters ware able to transfer RF tag data fron a 1.13.3 
handheld reader. defactve RF tag to he replacement RF tag. Occasionally, he 
HR would have some diffaulty fansiarring the data fo he 
ragiacement RF tag. 
The warighters stated the bamade labels were easy and qucker 
put asset data to fhe RF tac 





1.16 Tag Transmission. The RF tag 
arsmissions will be within HERO conpiant 
mdiatad power pararnaters. 


Lat2 
14a13 
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Requirements Results 
Data Tag 


0% and 95% relative humidity (non-condensing). The RF tags can 
function after a 15 Klovolt electrostatic discharge test and 4foot 
drap aio concrete. 

The RF tags operated for 10 days at Miesau and CAAA 
demonstration gdaced in the ASP. RF tags remained funcforal 
during the transfer fom CAAA to the Norfolk Naval Stafon via 


HHR. No major problems were observed during he 
demonstrations. 


At CAAA demonstrafon, ATOS operator was able to conduct an 
inventory of an earh-covered magazine within 20 minutes using 
‘fhe roll call function. The warfighter felt this functionality woud be 
a time saver in hie fidd. 

During fhe maritime demonstration, he HHR was effectve 
perfomning a speedy tally of 60 RF tags inside the USS Ponce 
10 22% & storage facility. The task was completed within 10 
minutes. 


On the USS Harry S.Truman, Det 1 AF OTEC performed a roll call 
operafon on 200 RF tags. The HHR was able to find 199 RF 
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Requirements 




























24 HHR to PP Interface. The HHR wit At the Miesau demons rafors, four HHRs tansfemed receipted 
communicate with he PP by mears of a data fo the PP during palet receipt operafors. The assesement 
direct connecfon (docking, remavatie media, | team compared the count of the handwritten inventory list fo the 
of infrared) with he PP server. count of the PP inventory list. This compar&an was 
accompished at al demonstafons. 
At the CAAA demonstafon, three HHRs transfered recaipted 
data to the PP during palet receipt operations. Note: The 
a@ssesenent ean observed techrical problems atthe CAAA 
demonstafon with the transfer of receipted data from the HHR 
fo the PP. 
On the USS Ponce, fee HHRs transfemed receipted data to he 
PP during pallet receipt. 
On the USS Harry S.Truman, free aut of four HHRs transfered 
receipted data to he PP during pallet receipt. Four HHRs were 
ued to perfoen fhe palet issue. The warfighters were able to 
update the pallet count and the assessment tear verified the 
pallet count changes by Visual inspecfion of the RF tags and PP. 
The warfightes stated the commurication and data transfer 
interface wif the HHR, RCU and PP needed some 
ro to increase fe : and accuracy. 
Not assessed. The HHR did not have removable data sterage 
meda. 







25 HHR Removable Memory. The HHR wit 

have removatie data stomge media such as 

flash memncry cards. 
iR 7 assessment team observed the HHR dd not werk if he 

RCU was interragaiing RF tags at he same fme. HRs wauld 

so lock up if anofher HHR was in clase proximity. 

HH was abe to 7 Nglel receipt, pale! 


arsacfon records, and change asset data an the RF tags. 
At the Miesau demors tafors, four HHRs ransfemed receipted 
data to the PP during palet receipt operafon. The assessmert 
team compared the count of the handwritten invertory list fo the 
count of the PP inventory list. This comparsan was 
accompished at al demonstafons. 

At the CAAA demonstrafon, three HHRs transferred receipted 
data to the PP during palet receipt operafiors. Note: The 
assessment ean obseved techrical problems atthe CAAA 
dermonstafon with he fanster of receipted data from the HHR 
fo the PP. 

On the USS Ponce, twee HHRs transfered receipted data fo he 
PP during pallet receipt. 

On the USS Hary S. Trurnan, tree out of four HHRs rareterred 
receipted data to the PP during pallet receipt. Four HHRs were 
ued to peffomn fhe palet issue during the mariime 
demonstafon. The warfghters were atie to update the pallet 
count and the assesernent tear verified the pallet court 






















change asset data on an existing tag. 


The HAR waremilier power was 240 mW. Halmel he 
requirements for HERO. 


2.10 HAR Future Design. Future design is The TECEVAL data reveded the HHR met the ruggedness 
for the HHIR to operate in any military requirements stipulated in he FRD. The assessmenttean 
environment worldwide. qbserved no HHR failure during the dernonstmfons. 
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Results 





Requirements 
RCU 


3.1 Installation. The RCU will be installed an 


ammunifion magazines, itis possitée that he 
RCU be mounted outside and the antenna 
cable run inside the magazine] The RCU 
may be installed using accesscry iterns, such 
88 a tipad, to enable the RCU to be installed 
in feid environments. 

antennas will be used for coverage of lage 


magazines and fied storage facilities if 
necessary. 


3.3 RCU Communications. The RCU will 
inifate hourly communications with the RF 
tags, collect tag ID, and any limitfiags, and 
transmit fis data fo fhe PP via a WLAN. in 
the event hata WLAN connecfon is not 
available from the RCU to the PP, the RCU 
will have the capability to stare 30 days of 
data from ail tags within its range and fansmit 
data when connectivity is restared. When a 
WLAN is not configured or not availatie, 
communicafon between fe RCU and the PP 
is achieved Wia direct docking wih he HHR. 
3.4 Power Configuration. The RCU shal 
operate fram on VDC. 


35 VW WN Operatio - o VLANS 
be used for data tmnamiasion between he 
RCU and fhe PP. 


3.6 RCU Multiple RF Tag Ant-collision. 
The RCU shail have an effective muttigie RF 
lag signal an$-colisicn feature. 

3.7 RCU HERO Compliance. The RC 
tranemissicns will be within HERO compliant 
radated power parameters. 


3.8 RCU Ruggedness. The RCU wil 
perfomn in a feld environment. To 

succes stully perform in the fed the RCU 
must be sealed to commerdal standard. 








indicated the lncaticn and access bility of the 
yea! Sie naoepeaiend ns nguyen infrastructure 
(conduit or wooden board) to hang the RCU and RFEs. Tie 
wraps were used to install the RFEs while he RCUs were 
mounted on a portatie stand cr tatie. 


demanstrations, te RCU was able to 
detect at least 98.5% of 200 RF tags during the first nterrogation 
cyde. During the second and third interragafon cydes, ATOS 
was able to detect 100% of he RF tags. 
ee ee eee 


RCU inlerrogated the RF tags every 60 minuies 
Demanstrations were ane or two weeks long. The RCU storage 
memay capacity was 256 MB. The assesenenttean did not 
chserve a data storage issue during the dencnstrations, but 
maximum capacity was not assessed. 


10.1 


131 
132 
13.19 


273 
275 
bx ey 


4s long a8 fe RCU was able to abtain 24 VDC power, he RCU 
was able to operate tawles aly. 
During the demorsrafion, the 24 VDC power saurce cane ton 
portable gas electric generator, High Mobily Mulipurpase 
Wheeled Veticle battery, of 120 VDC line power using a 24 VDC 
step down tanstormer. 
The CAAA logistics SME felt power availability at remote storage 
locations woud be a major concam. Many of these magazines 
will net have the infastucture to have dectrical power to 
iN ee 
entsbdebad with fe enthcavend magarte and PP: The 
distance between two locafons was approximately less han an 
aight of a mile. 
At he Crane dencstration, te WLAN was 
comected fo a repeater on top of a 75 foot water tower fo 
estabish a mile and half comnecton to he PP. 
At bom dermmstration sites, the WLAN was able to provide 

= fram te RCU to fhe PP. 
The assessment team obseved the HAR dd not work fe 
RCU was interragaing RF tags at the sarne Sime. 


The RCU mceived radiated energy fom the RF tags. The only 
RCU ransnission came fom the WLAN. The WLAN power 
radiated at 100mW Effective isatrapic Radiated Power, which 
maetihe HERO mq 

The TEC EVAL data revealed the RCU mat the ruggedness 
requirements sfpulated in he FRD. The assesementtean 
cbserved two RCU failures during the dernonstmations. Cause 
for fhe falures has not been determined. 
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Requirements Results 


4.1 PP Data Dissemination. The PP will 
make asset data available to Service 
amenurition AIS. Environmental data wil be 
available on the EDB at indan Head, MD, or 
over cenval sites. Envirawnental data will be 
sent to he EDB by way of secure (128-bit 


42 PP Data Visibility and Access: The PP ‘The wadighiers indicated that using he PP display was easy. 
Qneralars were able to abigin the inventory and transaction 
information needed to manage the tagged munifons pallets. 
‘The wadighters stated data entry cm the PP was easy using he 


will pass accurate data forn the RF tags to 
he PP. 


= 7 7 — 


reader will be detected by he PP within wo Gut of 200 RF tags (99.5%) while the ASP RCU detected 194 RF 
consecutive roll calls (appraxinaiely 145 tags (97. 1%). 

minutes based on haurly atlerngts) fram the At he Crane demonstration, te RCU was able to detect 197 out 
Start of the test. Data form 99% of he RF tags | of 200 RF tags (98.5%) inside he eat covered magazine and 
in he test magazines will be detected by he storage warehouse. 

PP within twee consecutive roll calls At he maritime demors¥aton a the USS Harry S.Trurman, he 
(approximately 215 minutes based on hourly RCU was aie to detect 200 out of 200 RF tag (100% ) inside 


atlenpts) torn the start of the test. he ship magazine. 
All 200 RF tags were picked up during the second interrogation 
cyde. 
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APPENDIX F 


CORRELATION OF 

ASSESSMENT QUESTIONNAIRES TO 
ENVIRONMENTAL SURVEILLANCE, 
Munitions MANAGEMENT, AND 
MEASURES OF PERFORMANCE 
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MOPs. Note that the Munitions QA task is not included in the table because it was not assessed during the 
demonstrations. 


Munitions Management 


Rate the acceptability of RF tag temperture sensor. 

Rate fie accephability of RF tag reiaive humidity senor. 

Rate fie miabiity of he RF tag seesors 

Rate ie capability to change he RF tag temperature limits from the made. 
Rate the capability to change the RF tag relativity lirit. 

Rate the efectiveress of RF taining. 

Rate fie efecivenest of HHR taining. 

Rate fhe efectveness of RCU taining. 

Rate fe efectversss of PP ranin 

Rate he efiectiveress of he ATOS training material. 


Rite fie Capability of Gie Re tags to execike an on comenand report, 

Rate ie capability of fe RF tags to execute an or-commnand memory eset. 

Rate the acceptability for veualy idenfitying the RF tag’s unique iden ication mum ber. 
of he RF tag barcode msactabilt 


Rate te acceptability of tansleming RF tag data to HHR and ROU. 

Rate te efectiveress of data tanefer fom HAR to PP_ 

Rate fie acceptability of tra: data from HHR of RCU to the PP. 

Rate fe acceptability of fie ATOS system to support data tansfer from Me HHR to he PP. 
Rate the acceptability of fe ATOS system to support data transfer from several RF tags to 
the RCU and PP. 

Rate the acceptability of fie antenna coverage between fhe RCU and fie RF tag. 


Rate the usability of the HHR stylus. 

Rate fie capability of fie HR to ceate new RF tags. 

Rate ie capability of fie HEIR to ceate replacement RF tags. 
Rate fie capability of fie HHR to cwate barcodes. 

Rate fie acceptability of fle RF tag afactenants to the containers. 


Rate he HHR data input usability. 
Rate tie usability of fhe HHF-RCU bridge. 
Rate fe usatilty of fe HHR-PP bridge. 
Rate fie usability of fe HHR-PP docking station. 
Rate fie usability of fe PP dispiay. 
Rafe he PP data inguh usability. 
bility of Cox AF tags. 

capita bility of cor te HAR. 
Rate fe acceptability of conSguiing fe RCU. 
Rate fie acceptability of confguiing fe PP. 
Rate fie acceptability of the RF tag power sequiremets. 
Rate fie acceptability of fie HHR and RCU power requirements. 
Rate cepta bility of PP po rene nts. 

= of fe RCU nstalation. 
Rate the acceptability of fie PP installation. 
Rate te acceptability of fie RF tag data storage memory capacity. 
Rate he acceptability of fie HMR ad RCU date storage memory capacity. 
Rate fie acceptability of fe PP data storage capacity 
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